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MEASURING PERFORMANCE: DEVELOPING GOOD 
ACQUISITION METRICS 


House of Representatives, 

Committee on Armed Services, 

Defense Acquisition Reform Panel, 
Washington, DC, Tuesday, May 19, 2009. 

The panel met, pursuant to call, at 8:00 a.m., in room 2212, Ray- 
burn House Office Building, Hon. Robert Andrews (chairman of the 
panel) presiding. 

OPENING STATEMENT OF HON. ROBERT ANDREWS, A REP- 
RESENTATIVE FROM NEW JERSEY, CHAIRMAN, PANEL ON 

DEFENSE ACQUISITION REFORM 

Mr. Andrews. Good morning, ladies and gentlemen. The panel 
will come to order. Appreciate your attendance. I especially want 
to thank the witnesses for their very diligent preparation. We have 
had the opportunity to review the testimony ahead of time. It looks 
like we are going to have a very engaging and meaningful discus- 
sion this morning. 

This is the last of our series of hearings looking at the first ques- 
tion that we are going to be looking at in our inquiry. The members 
will recall that we are proceeding on a series of questions, the first 
of which is, “Can we design a series of metrics that accurately 
measure the difference, if any, between the price paid by the tax- 
payers and the value received by the taxpayers and the warfighters 
for the systems and services that we are buying?” 

We are going to proceed after today’s hearing with our second 
category of questions, which is really hypotheses as to what has 
gone wrong. We are going to have a series of panels talk about 
their theories and analyses of why we have a difference between 
the price paid and the value received. 

The first of our two panels on this metrics question dealt with 
the sort of measuring the orthodox algorithms that are used. In the 
major weapons system side, we had a panel on that, and on the 
services side, we had a panel on that. 

The purpose of today’s panel is to bring in some people who we 
think perhaps look at the whole question through a different prism. 
They are willing to give us some new perspectives through which 
we can analyze the difference between the price we pay and the 
value that we receive. 

We have three witnesses with a wealth and breadth of experi- 
ence in the acquisition field, but also who I think benefit from what 
I would call a healthy distance from the daily responsibilities for 
that function, so they can give us a perspective that marries experi- 
ence with a fresh perspective. And hopefully, we will be able to use 
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the testimony of these witnesses to go forward, draw some conclu- 
sions about the best metrics to use in our work, and then proceed 
with our work in analyzing the various hypotheses given for why 
we have suffered these cost overruns. 

I also want to take a moment and thank Mr. Conaway, who can’t 
be with us this morning but who is ably represented by Mr. Coff- 
man, and all the members of the panel for their excellent contribu- 
tion to the work on the Weapons Acquisition System Reform 
Through Enhancing Technical Knowledge and Oversight (WASTE 
TKO) bill, which has passed the House. We will be meeting with 
the Senate in formal conference at 4:30 this afternoon to, I believe, 
put the finishing touches on that conference report so we will be 
able to proceed to the floor in short order on that, get the bill on 
the President’s desk. Each member of the panel has made a very 
significant and welcome contribution to that effort. 

Having said that, as the chairman and Mr. McHugh have said, 
we think that that effort covers about 20 percent of the landscape. 
It looks at major weapons acquisition, which certainly needed to be 
examined. But there are so many other areas that fall outside of 
that that the panel has to pursue, as well as, frankly, reviewing 
the early stage implementation of the law that we believe the 
President will sign this week. 

So we are by no means at the conclusion of our work. We are 
really at the outset of it. And one thing we would ask the witnesses 
to do this morning is to think about the fact that, although the 
Congress is about to pass major legislation dealing with major 
weapons systems, we have not yet addressed hardware that doesn’t 
fall into the major weapons system category and the 55 to 60 per- 
cent of acquisition that is services that we have to look at on behalf 
of those who wear the uniform and on behalf of the taxpayers. 

We are glad that you are here. And at this time, I want to turn 
to Mr. Coffman for his opening statement, and we will proceed with 
the witnesses after that. 

[The prepared statement of Mr. Andrews can be found in the Ap- 
pendix on page 27.] 

STATEMENT OF HON. MIKE COFFMAN, A REPRESENTATIVE 

FROM COLORADO, PANEL ON DEFENSE ACQUISITION RE- 
FORM 

Mr. Coffman. Good morning. Mr. Chairman, ladies and gentle- 
men, I would like to extend a welcome on behalf of Ranking Mem- 
ber Conaway, as well. He is sorry he could not be with us today. 
I would like to thank the chairman for allowing me to take a few 
introductory remarks in Representative Conaway’s place. 

Today’s hearing is an appropriate follow on to the panel’s two 
hearings which focus on how the Department of Defense (DOD) 
and the Government Accountability Office (GAO) currently assess 
performance on major weapons systems and service contracts. The 
purpose of this hearing is to think outside the box on how we 
should measure performance and about what we should be meas- 
uring and less about the how we are doing it today. 

Our previous hearings revealed that current measures of per- 
formance tend to break down if the program baseline is unrealistic. 
We want to know if the program can be corrected — if the problem 
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can be corrected. Furthermore, are there metrics beyond cost and 
schedule performance that are of value, such as how closely does 
the delivery system meet actual warfighter needs. Does the time of 
delivery of operational capabilities satisfy warfighter needs? How 
do we determine whether optional or tradable capabilities re- 
quested by the warfighter are affordable? 

Mr. Chairman, we have a distinguished panel of witnesses in 
front of us today. I have looked at their written testimony, and I 
look forward to their testimony. There are a couple of points I 
would like to highlight. Although Mr. Dillard is not testifying 
today, he did submit a written statement that I would like to brief- 
ly comment on. 

[The prepared statement of Mr. Dillard can be found in the Ap- 
pendix on page 77.] 

Mr. Coffman. He states that, “The proliferation of autonomous 
fiefdoms within the department continues to increase, with each 
being a stovepipe of oversight expertise, imposing unique reporting 
requirements, assessments and reviews.” 

He goes on to state that, in regards to adding additional acquisi- 
tion workforce professionals, that, “These new people should not be 
housed in the Pentagon, but instead where the execution of pro- 
grams occur.” Mr. Chairman, I think this is an important point 
that I believe we need to follow closely. 

Another point of observation I would like to make is in regards 
to requirements in joint programs. We have many programs out 
there that have the word “joint” in front of them, but they are joint 
in name only. I understand that the current requirements genera- 
tion process is called the Joint Capabilities Integration and Devel- 
opment System, or JCIDS. 

It seems to me that, in order to have a truly joint program that 
the requirements for that program must be borne jointly. Yet, I do 
not believe our current system, or as Mr. Dillard describes, current 
fiefdoms, foster such an environment. So I would be interested in 
hearing from our witnesses in this regard. 

I also encourage our witnesses to share their views on existing 
laws and regulations that are particularly helpful or not helpful to 
the Department’s efforts to obtain the best value and capability for 
our warfighters. We have heard from the Department in two sepa- 
rate hearings about how they currently measure performance and 
value on contracts. What we haven’t heard enough about is how 
they should be measuring value. Consequently, your input will be 
greatly appreciated. 

With that, I conclude. And again, thank you, fellow members, 
and thank you, Mr. Chairman. I look forward to the witnesses’ tes- 
timony. 

Mr. Andrews. Mr. Coffman, thank you. 

Without objection, the statement of any other member of the 
panel will be included in the record. And without objection, the 
statement of each of the three witnesses, plus the supplemental 
floor statement, will be included in the record of the hearing. 

[The prepared statement of Mr. Coffman can be found in the Ap- 
pendix on page 28.] 

Mr. Andrews. I think each of you are veterans at this process. 
You know that we generally have a five-minute rule where we ask 
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for succinct summaries of your written testimony. We are going to 
be a little bit more liberal with that this morning, but then we are 
going to turn as quickly as we can to the questions from the panel 
and get into the give-and-take. 

I am going to read a brief biography of each witness, and then, 
Mr. Patterson, we are going to begin with you. 

Mr. Dave Patterson is the Executive Director of the National De- 
fense Business Institute, which he is establishing at the University 
of Tennessee in the College of Business Administration. It is an in- 
stitution inspiring business innovation for both government and in- 
dustry by providing practical, sound assistance and creating eco- 
nomically efficient and effective defense business and acquisition 
programs. 

Prior to taking his current duties, he was the Principal Deputy 
Under Secretary of Defense Comptroller. As the Principal Deputy, 
he was directly responsible for advising and assisting the Under 
Secretary of Defense with development, execution and oversight of 
the DOD budget exceeding $515 billion, with annual supplemental 
requests of more than $160 billion. 

From August 2003 to June 2005, Mr. Patterson held duties as 
the Special Assistant to the Deputy Secretary of Defense. In that 
capacity, he was responsible for managing the Deputy Secretary of 
Defense’s personal staff, as well as providing direction and advice 
to the Office of the Secretary of Defense staff on a wide range of 
national security operations and policy subjects. 

He served in the Air Force from 1970 to 1993, retiring in the 
rank of colonel. During that time, he held responsible leadership 
and management positions, with assignments at the air wing level 
as the C5-A aircraft Commander and Deputy Operations Group 
Commander at Major Command Headquarters, U.S. Air Force, the 
Office of the Chairman, Joint Chiefs of Staff, the Office of the Sec- 
retary of Defense and Inspector General. In 1986, he was the Air 
Force Fellow at the American Enterprise Institute and served in 
Vietnam, flying 02-As as forward air controller. 

Thank you for your service to our country, Mr. Patterson. Glad 
to have you with us. 

Mr. Fitch, Mr. David P. Fitch enlisted in the Navy in 1966. He 
was commissioned in 1968 after graduation from San Jose State 
College and Aviation Officer Candidate School in Pensacola, Flor- 
ida. 

In 1969, after fixed and rotary wing flight training, he was des- 
ignated a naval aviator and a distinguished naval graduate. He re- 
tired from the Navy in 1998 as a captain, following a career that 
included three operational and major acquisition command assign- 
ments. 

Mr. Fitch culminated his tour in the Navy as the major program 
manager for the international and joint multi-functional informa- 
tion distribution system. He was a 1997 recipient of both the David 
Packard Excellence in Acquisition Award and the Department of 
Defense Value Engineering Award. 

In 2001, after nearly three years in the defense industry, Mr. 
Fitch joined the faculty of the Defense Acquisition University, 
teaching and consulting acquisition executives and program man- 
agement and other acquisition disciplines. 
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In 2006, he led a major independent study of the Coast Guard 
Deep Water program. In July of 2008, he assumed the position of 
Director, Acquisition, Technology and Logistics (AT&L) Leadership 
Learning Center of Excellence after nearly seven years as the Dean 
of the Defense Systems Management College. 

Mr. Fitch holds degrees in business and industrial management 
from San Jose State college, an M.S. in education from the Univer- 
sity of Southern California, and he is a graduate with highest dis- 
tinction from the Naval War College in Newport, Rhode Island. 
Thank you, Mr. Fitch, for your service to our country, and we are 
glad you are with us this morning. 

Dr. Daniel Nussbaum, from 2004 to the present, has been a vis- 
iting professor at the Naval Postgraduate School operations re- 
search department. From 1999 to 2004, he was a principal at Booz 
Allen Hamilton, responsible for a broad range of costs, financial 
and economic analyses with clients across the government and 
commercial spectrum. 

From 1996 to 1999, he was the director of the Naval Center for 
Cost Analysis at the Office of the Assistant Secretary of the Navy 
for Financial Management and Comptroller here in D.C. From 
1987 to 1996, he was division head of the Naval Center for Cost 
Analysis. From 1982 to 1986, he was Deputy Director and Acting 
Director for operations research and cost analysis divisions of the 
Naval Air Systems Command in Washington, D.C. 

Again, a very distinguished servant of our country, three excel- 
lent people with experience and fresh insight. We are happy to 
have you with us. 

Mr. Patterson, we would like you to begin with your testimony. 

STATEMENT OF J. DAVID PATTERSON, USAE (RETIRED), EXEC- 
UTIVE DIRECTOR, NATIONAL DEFENSE BUSINESS INSTI- 
TUTE, UNIVERSITY OF TENNESSEE 

Mr. Patterson. Thank you, Mr. Chairman, members of the De- 
fense Acquisition Reform Panel. I am very pleased to be here this 
morning to participate in a discussion of a question that has clearly 
captured the attention of the current Administration and Congress: 
How should Congress assist the Department of Defense in improv- 
ing its acquisition of weapons and services so that it can meet the 
needs of the warfighter in the field while still being a good steward 
of the taxpayer’s dollars? 

The first consideration for judging the success of an acquisition 
program is whether it fielded a weapons system or information sys- 
tem or service in time to make a positive impact for the warfighter. 
A system or service fielded too late to need may as well not have 
been bought at all. The phrase, “too little, too late,” can mean lost 
lives. 

But before we look at the measures of the acquisition system 
merit, there is another consideration central to this discussion. 
When Secretary Gates made his budget announcement on April 
6th, 2009, I believe he was speaking from frustration that was as 
much about what has been the persistent problem with the acquisi- 
tion system that we depend on that is simply not responsive to the 
immediate warfighter needs, as it was about winnowing bloated, 
failed and unnecessary programs. 
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Implicit in that expression of frustration is a clear lack of con- 
fidence in a system that actually produces programs with uncer- 
tainty and instability. The most dramatic improvement metric will 
be when the senior leadership in the Administration, Congress and 
the Department of Defense have seen such improvements, results, 
not words, that they can say they have renewed confidence in the 
stability, predictability and effectiveness of the defense acquisition 
system. 

The Defense Acquisition Performance Assessment’s Report con- 
tended that program stability and predictability were singularly 
and uniquely crucial to managing programs that were on cost, on 
schedule and performing. To that end, in the time I have, allow me 
to describe two areas of improvement for measuring program effec- 
tiveness worthy of attention. 

First, Major Defense Acquisition Programs, or MDAPs, often 
start at milestone B, the beginning of engineering and manufac- 
turing development, with critical staff positions vacant. Percentage 
of critical staff positions filled at milestone B is an easy and impor- 
tant metric to be observed. It makes little difference to implement 
programs to raise the level of skills of the program staff if they are 
missing in action. 

Second, the acquisition strategy document, that is to lay out how 
the weapon system is to be acquired, the initial roadmap, if you 
will, is often flawed in that it focuses more on presenting a case 
for required capabilities and quantities than on laying out the rea- 
soning for acquisition competition methodologies. 

For example, how the prime contractor participants in an MDAP 
competition will select subcontractors and how the winner of the 
competition will manage the subcontractors to gain improved effi- 
ciencies and effectiveness are generally given little consideration. 
Creation of the acquisition strategy document is one of, if not the 
most, important tasks the government acquisition program man- 
agement can undertake. 

The strategy should establish the template for all the activities 
that will take place throughout the source selection process, engi- 
neering manufacturing development, and follow-on production and 
fielding. More important, it establishes how the program manage- 
ment team is thinking about the numerous events and activities 
that a program will encounter. 

The defense acquisition executive should establish a common set 
of strategy elements that all the military department service acqui- 
sition executives must include in MDAP acquisition strategy docu- 
ments. Additionally, a set of standards or metrics by which the 
strategy elements can be evaluated as effective must be part of 
that process. 

In closing, I would be remiss if I didn’t acknowledge the progress 
that has been made by the Department in improving the acquisi- 
tion system over the last four years. Though it is the General Ac- 
countability Office’s headline that the 96 major acquisition pro- 
grams have grown in cost by $296 billion that gets attention, those 
numbers belie an equally worthy but overlooked statistic published 
in the very same report. 

The average increase in unit cost of the 28 MDAP programs with 
less than 5 years since development started is only 1 percent. Com- 
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pared with an average of 55 percent increase in acquisition unit 
cost of 25 programs in the group with 5 to 9 years since program 
development start, there has been important improvement that 
should be recognized. 

With that, it is my privilege to be here. Thank you very much, 
and I would be happy to take any questions that you have. 

[The prepared statement of Mr. Patterson can be found in the 
Appendix on page 32.] 

Mr. Andrews. Thank you, Mr. Patterson. 

Mr. Fitch. 

STATEMENT OF DAVID P. FITCH, USN (RETIRED), DIRECTOR, 

AT&L LEADERSHIP LEARNING CENTER OF EXCELLENCE, 

DEFENSE ACQUISITION UNIVERSITY 

Mr. Fitch. Chairman Andrews, Congressman Coffman, members 
of the panel, thank you for the opportunity to appear here today. 
I will address the subject of acquisition performance metrics and 
your questions about how to increase the realism of program base- 
lines, making trades between affordability and performance and 
how to assess the value of systems that do not necessarily — please 
recognize that my opinions do not necessarily reflect the views of 
the Defense Acquisition University (DAU), the Department of De- 
fense or the Administration. 

Mr. Andrews. That is why we are interested. 

Mr. Fitch. I suspected as much. 

Measurement of acquisition performance must encompass both 
strategic and tactical elements. As emphasized in a recent Defense 
Science Board report titled, “Creating a DOD Strategic Acquisition 
Platform,” the management, execution and oversight of acquisition 
programs is moot if we aren’t spending taxpayer dollars to buy the 
right capabilities if we aren’t demonstrating strategic choice. It is 
as important to decide what capabilities we won’t buy as well as 
what we will buy. 

I believe one of the root causes of funding instability is what I 
described as too many programs chasing too few dollars. Too many 
programs chasing too few dollars is one of the root causes, I be- 
lieve, of overly optimistic cost estimates. The recently implemented 
Materiel Development Decision (MDD) process provides a frame- 
work for making strategic investment decisions. 

The MDD is the formal point of inquiry into the acquisition proc- 
ess. MDD will increase integration of the three major acquisition 
support systems, requirements, resources, and acquisition. 

Improving requirements management, an initiative supported by 
the Congress, includes providing training to requirements writers 
and managers to ensure that they have a sufficient understanding 
of the critical elements of acquisition, such as systems engineering 
and testing. The goal is to improve collaboration between acquisi- 
tion and requirements communities to exploit cost and performance 
trades and improve acquisition outcomes. 

Having a formal requirements and acquisition process is not 
unique to the DOD. We can learn important lessons from commer- 
cial industry. If you compare the DOD acquisition system with the 
process to develop electronic games, there are marked similarities 
and differences. 
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Notably, the process to get games on the shelves for the Decem- 
ber holiday season starts with a precise clarity of what will be de- 
veloped by when, and includes a corporate commitment to the re- 
sources required for the project. Precise clarity is a result of in- 
tense interaction between the people that divine the capabilities of 
the game and the people that will design and test the software. 

Turning to the subject of tactical acquisition metrics, the most ef- 
fective tools and templates incorporate metrics, both quantitative 
and qualitative. The question was raised, “Are there metrics be- 
yond cost and schedule performance that are of value?” The answer 
is yes, and an ongoing example is a probability of program success 
metrics, POPS, that are currently being developed and deployed 
across the services and other federal agencies, such as the Depart- 
ment of Homeland Security. 

Starting with a blank sheet of paper, a group of DAU faculty 
comprised of experienced program managers and other functional 
experts ask themselves questions, including, “What conditions fa- 
cilitate the success of programs? What metrics are leading indica- 
tors of derailment?” 

The resulting tool, POPS, uses a structured process to contin- 
ually assess and display key elements of planning, resourcing, exe- 
cution and external influences that promote or negatively impact 
program success. Still evolving, POPS is being used in the Army, 
Air Force, Navy, Marine Corps and Coast Guard. Timely, accurate 
and transparent metrics integrated in a management and oversight 
process, will produce better program outcomes. 

Another question that has been asked, “Can we ensure improved 
realistic baselines.” Again, I believe the answer is yes, and there 
are ongoing initiatives that will yield more realistic baselines. 
These include increased emphasis on technology readiness before 
starting major development, emphasis on improved cost estimating, 
which I am sure that we will talk about today, and competitive 
prototyping, which is included in the new DOD 5000.2 and pending 
legislation. 

Prototyping increases the opportunity to identify and assess af- 
fordability and capability traits. Competitive prototyping also al- 
lows the government to observe the performance of competing in- 
dustry teams before making a down-select for engineering and 
manufacturing development. No matter how thoughtfully we plan 
and discipline source selection, a paper-only source selection proc- 
ess is never as good as demonstrated performance. 

The ultimate assessment of whether we have delivered value and 
needed capability to the warfighter is feedback from the field from 
the warfighter. At various times, I have seen photos of messages 
written with felt-tip pens on trucks and personnel carriers that 
have been returned to the government or industry depots. One of 
those signed messages — a picture is in my written testimony — 
reads, “This truck saved my life, as well as five others, 2-April- 
2008 at 2300 Lima, Basra, Iraq.” Those kinds of testimonies, those 
kinds of results, are what those of us that are in the acquisition 
aim to achieve on a daily basis by our efforts and commitment. 

Before equipment is fielded, it undergoes rigorous levels of devel- 
opment and operational testing, and the new DOD 5000 has in- 
creased emphasis on earlier testing. 
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Mr. Chairman, thank you very much for the opportunity to par- 
ticipate in today’s discussion. As the Secretary of Defense has said, 
there is no silver bullet. It will be a combination of initiatives and 
collaboration amongst all the stakeholders to create an acquisition 
system that consistently produces successful acquisition programs. 

I look forward to your questions. 

[The prepared statement of Mr. Fitch can be found in the Appen- 
dix on page 57.] 

Mr. Andrews. Mr. Fitch, thank you very much for your testi- 
mony. 

Dr. Nussbaum. 

STATEMENT OF DR. DANIEL A. NUSSBAUM, VISITING PRO- 
FESSOR, DEPARTMENT OF OPERATIONS RESEARCH, NAVAL 

POSTGRADUATE SCHOOL 

Mr. Nussbaum. Mr. Chairman, distinguished members of the 
panel, I would like to thank you for this opportunity to discuss my 
thoughts on how to improve acquisition and cost estimating proc- 
esses in the Department of Defense. These ideas are mine alone. 

As the chairman said, I am a member of the faculty at the Naval 
Postgraduate School in Monterey, California. And I have spent the 
last 30 years mainly doing, and more recently, teaching and re- 
searching in the defense acquisition management system with a 
focus on cost estimating. I was a previous director of the Naval 
Center for Cost Analysis and past president of the Society of Cost 
Estimating and Analysis. 

All my experiences in cost estimating confirm that three things 
are necessary for sound cost estimating: Acceptance of the under- 
lying uncertainties in predicting the future, accurate and plentiful 
historical data, and professionally trained and certified personnel. 

On uncertainty, there is intrinsic uncertainty in all estimates. It 
derives from several sources, mainly that we are usually designing 
or building or operating something that is substantively different 
from what we did before. The difference could be in the product or 
in the economic conditions or the programmatic conditions. 

An estimate reflects our knowledge at a point in time when we 
freeze the problem and base the cost on the configuration and 
programmatics as they are understood at that time. From that 
baseline, many things can change that can also change the cost es- 
timate, including labor rates, overhead rates, schedules, enhance- 
ments of the capabilities or quantities, changes when a particular 
technical solution to a problem doesn’t work as planned. And we 
need an alternative technical solution. 

On data, a hallmark, a necessary characteristic of a sound cost 
estimate, is that it is based on historical program performance from 
similar or related ongoing or past programs. Historical data is vari- 
able. Not every aircraft costs the same as every other aircraft. 

And the measurement of this variability is accomplished through 
statistical constructs, things like standard error of the estimate, 
things like confidence intervals. We assume in our community of 
cost estimating that the patterns of the past will repeat in the fu- 
ture. But these patterns are almost always statistically grounded 
patterns modeled with the powerful and subtle techniques known 
collectively as regression. Further, we know of no alternative ap- 
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proach to using the past as a guide to the future if we want a sci- 
entific that is a reproducible and auditable approach. 

Not all estimating is done by government employees. There real- 
ly are three sub-communities. One is the government in-house esti- 
mators. One is employees of the large vendors who also design, de- 
velop and build what we buy — Boeing, Northrop, Lockheed Martin, 
for example. And thirdly, there are support contractors or consult- 
ants to the government. Those are the three communities. 

And surely, we need to increase the capacity and the quality, the 
numbers and the training of the government estimators, but so do 
we need to enhance the professionalism of the other two commu- 
nities. 

There are currently no undergraduate curricula in cost esti- 
mating, and there are only four educational institutions that I am 
aware of that teach at least one course in cost estimating, and 
those are the Naval Postgraduate School in Monterey, where I am; 
the Air Force Institute of Technology at Wright-Patterson in Ohio; 
Defense Acquisition University, diverse locations with a capital 
campus at Fort Belvoir; and the Massachusetts Institute of Tech- 
nology, which offers one elective course in cost estimating within 
its engineering curriculum. 

The recent separation of the business cost estimating and finan- 
cial management career field into two separate cost estimating and 
financial management tracks is a very welcome development and 
should be supported, but note that DAU support is largely limited 
to military and DOD, not the other two communities. 

The Society of Cost Estimating and Analysis — we say SCEA — 
whose membership includes approximately one-third of all cost es- 
timators supporting DOD, is a central and indispensable player in 
the training, initial certification and periodic recertification of cost 
estimators. I note with pleasure that the executive director of 
SCEA, Mr. Elmer Clegg, is in the room. 

SCEA has collected a body of cost estimating knowledge, and it 
provides the members of the cost estimating community, provides 
training in cost estimating, has developed and offers an examina- 
tion and experience-based certification program. Lockheed Martin, 
Boeing, Northrop Grumman, other vendors use SCEA’s training 
and certification as their standard. 

I appreciate very much what this committee seeks to accomplish, 
Mr. Chairman. This concludes my prepared statement, and I would 
be pleased to answer your questions. 

[The prepared statement of Dr. Nussbaum can be found in the 
Appendix on page 66.] 

Mr. Andrews. Well, thank you, Dr. Nussbaum, and I thank each 
of the three witnesses for their testimony. As I say, we have had 
the chance to review the written testimony. We are now going to 
get to the questioning phase. 

Mr. Patterson, I was intrigued by your reference in your written 
testimony to two kinds of requirements costs, which you express as 
customer requirements and derived requirements. What is the dif- 
ference between the two? 

Mr. Patterson. Customer requirements are established in key 
performance parameters, which are requirements that are estab- 
lished for a particular weapons system that the weapons system 
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then must perform against. Derived requirements are those re- 
quirements that the customer did not ask for but that, as the name 
would suggest, derive as a consequence of the design process. “Oh, 
look, we could do this better if only we” — and of course, that is 
taken to the program manager. 

The program manager will say, “Okay, we can do that. Just 
bring money.” And they do. 

Mr. Andrews. Now, I notice on page seven of your testimony 
that you say that a recent study prepared by the Monitor Company 
Group was based on selected acquisition report data, estimates that 
approximately 33 percent of the cost growth from 2000, 2007 — I 
think I read this — is attributable to this second category of require- 
ments that you are talking about, right, the derived? 

Mr. Patterson. That is correct. Yes, sir. 

Mr. Andrews. Now, how is that number reached? Where did that 
33 percent come from? 

Mr. Patterson. That comes from the selected acquisition reports 
(SARs). In fact, they lay that out in the SARs and explain in each 
of the SARs where the 

Mr. Andrews. Well, one of the things that we want to do is to 
make sure that we discriminate between derived changes that are 
beneficial and those that may be superfluous. How would you sug- 
gest that we might do that? In other words, I don’t want to leave 
the impression that we are saying, or the report is saying, that 
“Oh, that 33 percent was waste.” 

Mr. Patterson. Oh, absolutely not. 

Mr. Andrews. And how do we draw the line between beneficial, 
cost-effective derived requirements and not-so-beneficial derived re- 
quirements? 

Mr. Patterson. Well, let me give you a real-life example. In 
1993, I came to the C-17 program. It was a troubled, problem- 
plagued program. 

The two program managers decided that much of the problem 
was that requirements were growing in an airplane. They estab- 
lished a set of rules. The set of rules was very simple: If you have 
a requirement or an engineering change, which is effectively a de- 
rived requirement, then it must go to an engineering change board. 

The engineering change board will evaluate the change for its in- 
trinsic merit. But if it doesn’t meet the safety of flight or other 
driven requirements, then it has to have a three-to-one payback in 
savings and not perturbate the schedule. 

Mr. Andrews. Interesting. 

Mr. Patterson. Real simple. 

Mr. Andrews. Now, in terms of the scope of that C-17 program, 
how much were the derived requirements overrun? In other words, 
how much was attributable to derived requirements? 

Mr. Patterson. I would only be guessing. I can certainly get 
that for the record. 

[The information referred to can be found in the Appendix on 
page 83.] 

Mr. Andrews. And with this method that you just described, in 
your judgment, what percent of the derived requirements were ben- 
eficial, and which failed to make that three-to-one cut and didn’t 
happen? 
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Mr. Patterson. Again, let me give you an example. 

At Edwards Air Force Base, they were testing the airplane. One 
of the requirements was that one airplane needed to start another 
airplane, but it started it with two hoses from the pneumatic sys- 
tem. Somebody said, “I wonder if it will start with one hose.” It 
didn’t. “Oh, my gosh. Well, now we have to go and figure out why 
that is the case.” 

And so, that took a considerable amount of test time and money, 
and as it turned out, they really wanted to have it start with one 
hose. I am sorry, that would have taken 

Mr. Andrews. Who was the someone who said that? Not the per- 
son’s name, but where were they in 

Mr. Patterson. That was the test community out on the ramp 
at Edwards. 

Mr. Andrews. Right. 

I am going to go back for a second round, but Mr. Fitch men- 
tioned this probability of success metrics program. Has there been 
built into that a litmus test for a probability, if you would fall 
below it, things stop? In other words, does that program have in 
it a built-in go or no-go line? 

Mr. Fitch. It does not have a go or no-go line. It is information 
that is updated and provided 

Mr. Andrews. Do you think it should? 

Mr. Fitch. I think that there needs to be an informed review and 
decisions that look at each of the things that occur that is negative, 
because some of the things that are in the probability of success 
metrics and reporting are positive. 

So I think that it needs to have the program manager, when that 
information is put together on a monthly basis, needs to look at 
that. It needs to be reviewed by someone in his Program Executive 
Office (PEO), okay, and potentially even at higher authority when 
there are negative occurrences. 

Could I add something about — reporting? 

Mr. Andrews. Sure, and then we are going to go to Mr. Coffman. 
Sure. 

Mr. Fitch. Yes. 

I just wanted to explain that the derived requirement process is 
actually part of the systems engineering process so that if, for in- 
stance, the warfighter says in an aircraft, “I need a display that 
has color, I need a display that has this amount of resolution,” et 
cetera, the first pass, even by the warfighter or the cost estimators, 
may say, “This amount of processing may be sufficient for that.” 

When you get into, “Well, by the way, you are going to have 
these other software capabilities.” You put those together, and they 
start to build on one another, you can find all of a sudden that the 
processor that was planned may be a commercial off-the-shelf item 
with X amount of memory, throughput, et cetera, is insufficient. 

At that point is the point where it would be very useful to have 
the requirements community have a real dialogue with the acquisi- 
tion community to say do I now take away some of the require- 
ments so I can continue to use that commercial off-the-shelf proc- 
essor that is less expensive, or are those requirements really impor- 
tant now that we have figured out what they are. So the point is 
the derived requirements is a very important process, and it is ac- 
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tually the process where you allocate where features will go to 
hardware or software. 

Mr. Andrews. Thank you very much. 

Mr. Coffman. 

Mr. Coffman. Thank you, Mr. Chairman. 

This is to all of you. It seems like part of the problem is some- 
times we are dealing with immature technologies, and I think we 
have had previous testimony to that effect, that sometimes we are 
asking the contractors to develop something. 

Should we bifurcate the process? In other words, that you are 
contracting with one entity to develop the technology, to develop 
the — I don’t know if you would go as far as a prototype, and then 
where you can go into fixed cost production with another — maybe 
that entity would be allowed to bid on it as well. But is it better 
to bifurcate the process? 

Mr. Nussbaum. You know, I would say we would do that now. 
If we have open competition, we certainly have a research and de- 
velopment (R&D) contract followed by an acquisition contract 
with — there is certainly no guarantee that the R&D contractor will 
have a follow on. 

So in some sense, we do that, but in another sense, the Depart- 
ment has now mandated technology readiness levels, TRLs of cer- 
tain levels before a program can get beyond a milestone. They have 
narrowed that cone of uncertainty by saying you have to have a 
TRL level six. They would prefer seven, and GAO prefers seven, 
but you have to pass a TRL level of six before you go into the next 
milestone. 

Mr. Fitch. As a part of the analysis of alternatives, that process 
that immediately follows it, the technologies that would be appro- 
priate for that system, that capability, are assessed for technology 
readiness, when you get to the point of saying who should develop 
it, I prefer the concept of the competitive prototyping, because you 
are going to get proposals from industry. 

They are not going to all have the same strengths. Some are 
going to view that I want to do this more in a hardware function. 
Others may be more software intensive. There are different tech- 
nologies involved. 

Their proposal, and then watching them deliver on what they 
promise and they say is possible in competitive prototyping, is key 
to then being able to transition and make an award to that con- 
tractor who has proven that they did what they say they could do. 
So having a different contractor, if I understood your premise cor- 
rectly, to develop technology and then award it to somebody dif- 
ferently to develop the system, I think that is not the intent of the 
competitive prototyping system. 

Mr. Patterson. I think, too, there is a problem that arises when 
we talk about spiral development. Originally, spiral development, 
evolutionary development, were designed to have a weapons system 
that effectively was fielded and then have block upgrades to im- 
prove the weapons system. 

In many cases, that is not how that works, because you do have 
a parallel R&D program that is working on improvements. And be- 
fore the weapon is actually fielded, you have the attempt to inte- 
grate improvements. 
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Well, the consequence of that often is that you have a stretched- 
out program. Costs escalate. And in the end, performance is de- 
graded, and it takes a while for the — much longer than had been 
anticipated for that weapons system to be fielded. So I think we 
need to have very specific and clear understood standards by which 
we will integrate or put in upgrades or technology advances into 
weapons system. It certainly can’t be during engineering, manufac- 
turing, development. 

Mr. Coffman. I think it was mentioned, and it has been men- 
tioned repeatedly, that the changes drive a lot of the cost as they 
go forward. Where did the changes emanate from? I mean, are 
some of the changes sort of broader in scope where the affected 
branch who will receive and utilize the weapons system, is it 
changes in evolving doctrine, or is it mostly at a very technical 
level with this engineering requirements? 

Mr. Nussbaum. You know, I think that changes come from all 
sorts of places, including changes in labor, overhead rates, sched- 
ules, requirements, quantity, absolutely everything, just like build- 
ing a house. Everything that changes has the contractor saying, 
“Cement has gone up. Brick has gone up. My subs have gone up” — 
or down, but they are always changing. 

And so there is a great churn. Some of it is part of life, and some 
of it we try to control by saying, “Tell us what you are building, 
and we will cost that program.” But I think it is just intrinsic in 
the cost estimating process. 

Mr. Coffman. Is there any way to just have more — I mean, what 
would be some of the methods for having more discipline over the — 
and I take it some of it would — having a change board? 

Mr. Nussbaum. There is something called a CARD, a Cost Anal- 
ysis Requirement Document, which is, to the chagrin of all program 
managers, we say three or six months before we are going to a 
milestone, you tell us what you are building, how many, what the 
technologies are, and we will cost that program. It may be that 
things change after that, but that is the program we are going to 
cost, because otherwise we are chasing our tail, and we don’t know 
until the very last moment what we are costing. 

And in fact, it takes time to do a cost estimate. It is a part of 
the discipline at the Office of the Secretary of Defense (OSD) is a 
Cost Analysis Requirements Document will be prepared six months 
before the milestone, and the lack of availability of that CARD re- 
sults in a day-to-day slip in the milestone. So that is real discipline, 
but it has its obvious down side, too. 

Mr. Patterson. I think, too, that in dealing with the discipline 
and structure, that I must tell you, I submit that it requires rules. 
And the rules simply have to say that, after milestone B, you don’t 
have any more requirements. And, I mean, it is somewhat draco- 
nian, but nonetheless, if it — unless of course it is a safety of flight 
or it is an obvious design failure that needs to be corrected. 

But those kinds of things are few and far between, quite frankly. 
And then, this idea that a requirement or a change would give you 
a three to one or four to one payback in savings while not per- 
turbing the schedule is not a bad idea, either. And as technology 
moves forward, that is entirely possible over the course of a pro- 
gram. 
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Mr. Fitch. There are times with the requirements — I recall when 
I was developing a black box that was going to go in 17 different 
platforms. I would have this platform coming to me and say, “It 
would really be easier if you changed it this way. It will make me 
less expensive for my integration, save my costs and schedule.” 

What I came to understand is the program manager has to be 
prepared to say no during the development phase. Now, the con- 
figuration steering boards we have in place anticipates that there 
may be compelling reasons to change a requirement, moderate a re- 
quirement during development, but that is the process of the con- 
figuration steering board, is to raise that, if you would, visibility to 
the pressures of changing requirements to have a senior level deci- 
sion made about the requirement once the milestone has been ap- 
proved. 

Mr. Andrews. Thank you. 

Mr. Cooper. 

Mr. Cooper. Thank you, Mr. Chairman. 

And I would especially like to welcome a fellow Tennessean, 
David Patterson, and congratulate him on starting Defense Busi- 
ness Institute at the great University of Tennessee in Knoxville. 
Thank you not only for your past service, but what you are doing 
right now. 

Can anyone provide me with enough historical perspective to 
help me understand how, during World War II, men like Henry 
Kaiser were able to produce destroyers and — I forget exactly who 
produced airplanes, but it was an amazingly productive period. And 
I don’t know how — was there less bureaucracy then? How was so 
much able to be accomplished so quickly? Anybody know? 

Mr. Nussbaum. My reading of history is that there were a lot of 
failures, and then we tend to remember the successes, which were 
terrific. But in fact, there were false starts. 

The P-51 was wonderfully successful, but because it had a bigger 
gas tank so it could keep up with the other aircraft, it gave us 
greater range, but we didn’t build it for that reason. Just we sort 
of lucked into that, if you will. 

I think that things were much simpler. We were able to turn 
technology generations around much faster and, therefore, absorb 
the failures. Today’s systems are very complex. They take a long 
time, and we just won’t accept failure. 

Mr. Fitch. I think that what we remember a lot is those ships 
going down the ways, okay, daily, okay? That was in the production 
phase. 

Even the P-51 that was just mentioned by Mr. Nussbaum, when 
it was first fielded, it didn’t have the engine that ended up being 
what everybody remembers about the performance of that aircraft. 
There were various modifications made after it was fielded. It was 
in the field and found that it was not exactly was needed. 

Mr. Patterson. I think, too, you had a tremendous industrial 
base that was able to accommodate to the level of technology that 
it was asked to accommodate to. Today, I am not sure that we 
would be able to do that again in that amount of time. 

Dr. Ron Sega did a study not long ago when he was at DDR&E, 
Director of Defense Research and Engineering, in which he looked 
at the industrial base and what it could do, and found that 62 per- 
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cent of all of the Ph.D candidates that are enrolled in disciplines 
critical to national security have temporary visas. And that tends 
to put us at a disadvantage, because they generally don’t stay 
around and work for Lockheed or Northrop or Boeing, and Skunk 
Works, particularly, or Phantom Works. 

And if you will look at the Aerospace Industries Association data, 
you will find that in the 1990, 1991, you had 1.3 million touch 
labor workers, highly skilled workers. Today, we have something 
less than 700,000. Those are statistics that should give us pause 
to consider what we are going to do in the future. 

Mr. Cooper. Help me understand. I know that we have had fail- 
ures in acquisition throughout our history, and Mr. Fitch men- 
tioned a success when he posted the note that was on the vehicle 
that saved lives. 

But on the front page of the Washington Post recently was a 
statement by Secretary of Defense Gates when he attended the re- 
turn of the remains of some of our troops. He asked how they died, 
and he was told they were in an inadequate vehicle. And he cursed 
because that was symbolic of the fact that we have had difficulty 
fielding relatively simple platforms like Mine Resistant Ambush 
Protected vehicles (MRAPs) or up-armored Humvees. 

And at least for some period of time there was only one manufac- 
turer of up-armored Humvees in America, and yet, at the same 
time, we have automobile companies going bankrupt and looking 
for vehicles to build. So there seems to be a mismatch somehow be- 
tween pretty basic demands for troops and our ability to field and 
source those even when we have able and willing automobile com- 
panies who are looking for ways to keep their plants busy. Why the 
mismatch? 

Mr. Patterson. Oh, I think it is a very fundamental problem in 
that you have a Department of Defense that understands fully that 
we are at war with terrorists, and you have a country that doesn’t. 

Mr. Cooper. So Chrysler or General Motors (GM) or Ford didn’t 
want to bid on the up-armored Humvee, or we couldn’t 

Mr. Patterson. Well, actually, the subsidiaries of those folks 
did, but they really were not in the business of putting out those 
kinds of vehicles. The people who were, International Harvester, 
obviously, and — but the design, fortunately or unfortunately for us, 
that was available were foreign designs of up-armored vehicles that 
were designed to sustain the kinds of things that an MRAP would 
have to sustain. 

Mr. Fitch. And the number of companies that have as a core 
competency in the technologies for armor, okay, is not the same as 
we have for the auto industry. In fact, we are doing a lot of invest- 
ment in the Department of Defense today to try to find better 
armor, cheaper armor, especially lighter armor, whether that be for 
the vehicles because, when you put the armor on it, it puts a de- 
mand on the engine. It puts a demand on the drivetrain. If you 
double the weight of the vehicle, whatever the percentage is, okay, 
it has additional tolls upon the reliability of the vehicle. 

So we don’t have the same industrial base. In other words, the 
auto industry isn’t the industrial base for our Army, either. 
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Mr. Nussbaum. In a sense, the Humvee is the wrong vehicle to 
up-armor, but it is the only vehicle to up-armor. But it was built 
as a replacement for the Jeep, which didn’t go in harm’s way. 

So it was designed to optimize all the functions that it was going 
to perform, and we missed the fact that it was going to go in 
harm’s way. So it wasn’t suitable, wasn’t optimal, for up-armoring, 
but it is what we have, so we are going to up-armor it. 

Mr. Cooper. I see that my time has expired. Mr. Chairman, look 
forward to another round. 

Mr. Andrews. Thank you. 

With the consent of my colleagues, we are going to do another 
round, if that is okay with the panel as well, if it fits your schedule. 
Thank you. 

Mr. Patterson just made a suggestion that we might have a rule 
that says no new requirements after milestone B, and that prompt- 
ed this question. Again, this piece of data that you cite, about 33 
percent of the program cost growth being attributable to these re- 
quirements changes, how many of those requirements changes hap- 
pened after milestone B? Do you know? 

Mr. Patterson. That data, as I recall, is after milestone B. 

Mr. Andrews. It is all after milestone B. 

Mr. Patterson. That is where the majority of the growth gen- 
erally takes place. 

Mr. Andrews. A related question: in your oral testimony, you 
talk about the fact that, if I read this correctly, the average in- 
crease in unit cost of the 28 MDAP programs of less than 5 years 
since development is only 1 percent. 

Mr. Patterson. That is correct. That is 

Mr. Andrews. But it is 55 percent from years 5 to 9. How many 
of that in the 55 percent category, the five to nine, got a waiver 
through milestone B, didn’t meet the requirements to get to mile- 
stone B but got waived past it? Do you know? 

Mr. Patterson. No, I don’t. I don’t know exactly how many. 

Mr. Andrews. But, I mean, would it be accurate to say it is prob- 
ably most of the 55 percent cost overrun comes from that? 

Mr. Patterson. I would say a significant portion of it, sure. 

Mr. Andrews. One of the things that is in the conference report 
that we will be looking at in the WASTE TKO bill today is what 
we call intensive care, where if a program is permitted to go for- 
ward, even though it didn’t achieve the milestone B criteria, if it 
is waived past it, there is a whole set of intensive requirements 
that are imposed upon that program to try to get it back under con- 
trol. 

I wanted to come back to, again, this bifurcation that you create 
between derived and customer requirements. Describe for us the 
process that you think ought to be instituted to determine whether 
a derived requirement is added to the package or not. 

Let’s say we are at a point — assuming for a moment that we ac- 
cept your proposition that there are none after milestone B, which 
I assume you say there should be some exceptions, now as you 
said, for true safety or emergency purposes, but let’s assume we 
are living in a world where, except for those narrow situations, 
there are going to be no changes in requirements after we hit mile- 
stone B. 
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We are now in pre-milestone B, and an “Oh, by the way” comes 
up, as you said earlier, “Oh, by the way, this can do this.” Who 
should make the decision as to whether that gets added to the 
package, and by what criteria? 

Mr. Patterson. Well, I think that the program manager should 
have the initial cut at whether or not they are going to include that 
into the program. But program managers generally are colonels or, 
in very large programs, brigadier generals who have significant 
oversight within the Department. 

I think that if they have a set of rules that say, first of all, if 
it is not a safety of flight or if it is not some sort of safety issue, 
or if it doesn’t give me a return on my investment, then I am going 
to have a thumbs-down, initially. 

Mr. Andrews. How do you measure the concept of return? 

Mr. Patterson. Well, let’s take, for example, you have — again, 
I will turn to this C— 17. 

There were parts of that airplane that were originally designed 
for aluminum lithium, for example. Well, aluminum lithium is 
strong, but it is brittle. 

So an engineering change was made to change that to a different 
alloy. We were breaking the aluminum lithium cargo floor guides 
at a regular pace, so changing that eliminated the problem of hav- 
ing to constantly replace it. That was a savings. 

And those are the kinds of things that I would suggest are three- 
to-one. And even the suppliers were given the opportunity to do 
that. 

Mr. Andrews. Got you. 

It strikes me that this is really the essence of the 20 percent the 
secretary talks about in his 80 percent solution, that what he really 
is aiming to get at here is to give us an adjudicatory mechanism 
that draws the line between the 80 and the 20 when you get to this 
point, and it is your suggestion the program manager should be the 
first person to weigh in on this. 

Who should evaluate his or her recommendation? 

Mr. Patterson. Well, then you have an engineering change 
board that would provide a corporate view of it. And if it is a par- 
ticularly expensive change, then you are going to go to your service 
acquisition executive, or if it is an ACAT 1-D, an acquisition cat- 
egory 1-D, then you have the Defense Department acquisition ex- 
ecutive who would have a cut at that. 

Mr. Andrews. And my final question, are you confident that we 
can quantify this concept of value sufficiently to hit the three or 
four? In other words, are all of the values that we want to pro- 
mote — you gave a great example of saving, replacing a piece of a 
plane that is going to go wrong. But do all of the value concepts 
lend themselves to that kind of quantification that would let us 
say, “Well, this fails to meet three-to-one, so out?” Pretty hard? 

Mr. Patterson. No, it is much more difficult than that. And that 
is why it takes a lot of research and study and to set up standards 
and conditions whereby you can evaluate these. 

Mr. Andrews. Frankly, and I will just conclude with this, it is 
one of the reasons why we are glad we have the three of you and 
the institutions that you represent, because we really do turn to in- 
stitutions like yours to assemble those data, analyze them and give 
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us a factual basis to draw the lines that my questions imply. 
Thank you. 

Mr. Coffman, your turn. 

Mr. Nussbaum. Mr. Chairman, is it 

Mr. Andrews. Sure, Doctor. 

Mr. Nussbaum. Is it appropriate for me to make a remark, 
or 

Mr. Andrews. Sure. It is okay. 

Mr. Nussbaum. I think it is not hard to measure the value of 
ideas that replace current capabilities. It is always harder to meas- 
ure the value of things that represent new capabilities. They don’t 
represent a savings for the operating and support detail that you 
were going to incur. 

But if you are replacing a current capability, then it is pretty 
easy to do an estimate, but it is still an estimate, of what does it 
cost to invest to make this thing happen, and what do you save 
over time in the operating 

Mr. Andrews. I agree. The much more difficult proposition is 
where you have a new function that could be added by something 
that you discover. How do you measure that, and that requires 
trade-off analysis. It requires opportunity cost analysis. It requires 
a lot of broader inquiries. 

Mr. Nussbaum. And if you have that long tail, the question then 
is do you do any discounting on it, the technical issue of net 
present value and at what rates. The Office of Management and 
Budget (OMB) helps us there. 

Mr. Andrews. Not to be hyper-technical, but one of our problems 
is then matching up the federal credit scoring and net present 
value rules with the real-world ones, that very often a decision — 
a classic example is in energy. The Department has guidance to hit 
25 percent alternative fuels by 2020. And in order for them to do 
that, they need to do multi-year contracts. But to do multi-year 
contracts, the Congressional Budget Office (CBO) scores that as 
putting the whole net present value into one year, which makes it 
almost impossible to do, which means we don’t do much of it, which 
means we are falling backwards. So marrying the CBO criteria 
with the real-world criteria is a bit of a challenge, too. 

Mr. Nussbaum. Yes, sir. And so the devil really is in the details 
on this, and 

Mr. Andrews. Well, the devil is in the CBO in this case. Now, 
don’t tell Mr. Elmendorf I said that, okay? 

Mr. Nussbaum. And one other comment, and that is, when you 
go to the Configuration Control Board or your Service Acquisition 
Executive (SAE), you are proposing to spend investment dollars to 
make this thing happen, and the promise is that you will return 
operation and support (O&S) dollars later on. That is a nice con- 
versation, but it doesn’t accord with the budgeting realities. 

Mr. Andrews. And it doesn’t score in our — nor should it. 

Mr. Fitch, do you want to add one thing, then I am going to go 
to my friend from Colorado? 

Mr. Fitch. If I could. I just wanted to say, again, that I think 
it is useful to talk about the operation requirements, the derived 
requirements, but to say again that, when we get the requirement 
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from the user, it is stated usually in operational terms, okay, the 
results they wanted to see. 

Mr. Andrews. Yes. 

Mr. Fitch. To deal with industry, and for industry to build some- 
thing, those need to be translated into technical terms. We also use 
the term “derived requirements” for that process. So the derived re- 
quirements are really — there are a couple types we are talking 
about here, that which are a part of the normal system that you 
have to do. 

The other one I would just say is your question is how do we 
know we get value. As a program manager, I had an acquisition 
program baseline. That was my contract. The way I viewed it, it 
was my contract with my — decision authority and with the Amer- 
ican taxpayer to produce a capability at such a cost with certain 
milestones. 

And I think that is what most of us take and go back to. And 
as we do these questions about derived requirements and every- 
thing else, I think we keep in mind that framework. 

Mr. Andrews. Yes. I think the panel clearly understands that 
some subset of derived requirements are quite legitimate, nec- 
essary and desirable. And I think the three of you have given us 
some interesting tools to discriminate between undesirable derived 
requirements and desirable ones, which is what we are about. 

Mr. Coffman. 

Mr. Coffman. Thank you, Mr. Chairman. 

We talked today about having discipline in terms of change re- 
quirements. But do we also have to have discipline when programs 
get so far out of line that they become questionable? 

And I am thinking about — and I am working if you could reflect 
on the President’s — I can’t remember the nomenclature of the fol- 
low-on helicopter. Littoral Combat Ship (LCS) is in question. I won- 
der if you all could reflect on when a program gets so far out of 
line. 

Mr. Patterson. Well, it would be helpful, I think, in the total 
scheme of things, if we never let them get out of line. But nonethe- 
less, you are exactly right. They do happen. It does happen. 

And one of the things that I think is difficult when you take, for 
example, the VH— 71, the President’s helicopter, and that is a per- 
fect example of where you came in with one set of criteria as re- 
quirements and, over the course of the time, it changed dramati- 
cally. 

I think that we don’t have a set of standards or conditions that 
raise a budget flag that say, “Wait a second, I am sorry, you are 
red here, and you have been red three reviews in a row, and we 
are canceling your program.” And what we do is we put ourselves 
in a position where it is the only game in town. 

You have the President’s helicopter that is arguably old, and we 
don’t have an alternative. You chose a manufacturer and a heli- 
copter, and there is no off-ramp. There is no plan B. And we do this 
rather consistently. 

And I am almost of the opinion that we do it by design. And I 
would offer the Marines’ Expeditionary Fighting Vehicle (EFV), for 
example. We don’t have a way to walk the dog back down the path 
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to get an amphibious vehicle that would replace the EFV, so we 
need to make that work. 

Mr. Fitch. I don’t have the particulars on either of the programs 
you asked about. I was the systems engineer for the VH-3D, the 
current — well, it is one of the two current presidential helicopters. 
It is a unique mission. I think there is a desire usually if the White 
House says, or whatever or whoever it is, says, “I need a capa- 
bility,” that you find a way to do it. 

Going back to it, most of the changes that I saw, and the pres- 
sures and the surprises that I got, occur in that first — traditionally, 
the first 12 to 15 months of a program when you go towards a Pre- 
liminary Design Review (PDR), because the contractor is off doing 
a lot of things. There isn’t a lot of object deliverables to figure out 
does he get it, he doesn’t get it. 

I think that the competitive prototyping system to focus on doing 
PDRs earlier, to have effective communications with the contractor 
teams and oversight of the contractor teams during that period of 
time will do much more to get to a stable baseline at milestone B, 
which is the actual program initiation. 

Mr. Nussbaum. I am going to sound like a professor on one hand. 
On the other hand, because there are some historical examples of 
things which failed, failed, failed and then were terrific. Aegis was 
one, and Tomahawk was another. They just took a long time to 
bring aboard. And for some reason, we had the fortitude to stick 
with them and not say, “This is an A-12 or a Gama Goat,” and get 
rid of it. 

So now we come to the category of LCS. Is it a Tomahawk or is 
it a Gama Goat? We don’t know. But I am taken with my col- 
league’s remark that, once you get past milestone B, if you have 
three reds in a row, you have got some serious explaining to do, 
with the presumptive answer, “You are out.” It is a rule. 

But the problem is knowing the future, and that is always the 
problem. I don’t know whether LCS is a Tomahawk which is going 
to be absolutely terrific after a long incubation period, and the 
same for V-22. You just don’t know, so we make decisions as peo- 
ple. 

Mr. Coffman. Thank you. 

Thank you, Mr. Chairman. 

Mr. Andrews. Thank you. 

Well, gentlemen, thank you very much for your expertise. Your 
reward for doing such a good job is we will have to call on you 
again. As the committee goes forward, our intention is to try to 
make legislative proposals for the fiscal year 2011 authorization 
bill that will deal with the area of the problem that the WASTE 
TKO legislation that we are dealing with today does not deal with. 

And I think you have given us some very intriguing ways to 
measure the gap between what we pay and what we get. It is also 
heartening to hear what I have heard this morning, a consensus 
that this panel’s contribution to the WASTE TKO bill was essen- 
tially two concepts. 

The first was to add a whole series of reviews and a lot of scru- 
tiny pre-milestone B, with particular emphasis on the requirements 
process. And that did make it into the conference report, and that 
will become the law this week, we think. 
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And second, the panel was very interested in much more rigorous 
review, what we call intensive care, of systems that pass milestone 
B by waiver, that have not met the requirements, or that fail 
Nunn-McCurdy standards and get exempted from the penalties 
there and go forward anyway. And I think if you look at the cost 
overruns, a huge majority of them fall into one of those two cat- 
egories. 

So what we wanted to do was to take the best practices that you 
very ably described this morning and engage them as intensely as 
we can in the systems that, again, never met the criteria to get 
past milestone B but get past it anyway, and those that fail Nunn- 
McCurdy but continue to live on. 

And as Mr. Patterson said a few minutes ago — I think it was Mr. 
Patterson — our ultimate goal is not to have any of those cases in 
the future by unraveling the requirements process and looking at 
it more intensely to intensify that pre-milestone B analysis of what 
is going on. 

And the other point that I would make that is more on our side 
of the table than yours, I think that the principal reason that we 
get these cost overruns is that, once something passes milestone B, 
an enormous political constituency develops around it. Now, there 
are tens of thousands of people deriving their paycheck from a 
project, hundreds or thousands of subcontractors, dozens or hun- 
dreds of congressional districts. 

And as Secretary Gates I think can attest, making changes in 
those programs is very politically difficult. If you get to these 
flawed programs earlier when their political constituencies are 
smaller and weaker, the chance to do the right thing is a lot high- 
er. 

So not just for analytical reasons, but we think, given the dy- 
namic of the way these decisions are made in the political world, 
the more precise we are in our measurements and the more focused 
we are in our evaluation, in the requirements phase and a little bit 
beyond that, we think the better job that we will do. 

So I would say to each of the three of you we welcome your con- 
tinued participation and input. We are certainly going to call upon 
you for your feedback as we go forward in our drafting process. 
And thank you very, very much for your time and attention this 
morning. 

Members will have a period of time by contacting either majority 
or minority staff to supplement the record with written questions, 
and we would invite the witnesses to do the same thing. 

And with that, the hearing is adjourned. 

[Whereupon, at 9:09 a.m., the panel was adjourned.] 
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Ladies and Gentlemen welcome to today's hearing on Measuring Performance: Developing 
Good Acquisition Metrics. Today's hearing is the third in a series of hearings exploring how we 
determine if the Department of Defense is getting what it pays for in the acquisition system. We have 
with us today Mr. David Fitch, Director of the Acquisition Leadership Learning Center of Excellence at 
the Defense Acquisition University and former Dean of the Defense Systems Management College. Mr. 
David Patterson, Executive Director of the National Defense Business Initiative, former Deputy Under 
Secretary of Defense (Comptroller) and former Executive Director of the Defense Acquisition 
Performance Assessment, and Dr. Dan Nussbaum of the Nava! Postgraduate School, former Director of 
Cost Analysis for the Navy. 

In our previous two hearings, we've learned that there are a lot of pitfalls and shortcomings in 
the Department of Defense's systems for measuring performance in acquisition. The Department's 
current metrics are highly susceptible to error, whether it is because the baselines that serve as the 
basis for tracking programs are unrealistic; because we measure individual programs in isolation rather 
than the system as a whole; or because metrics are still in their infancy for the 80% of the acquisition 
system that is not major weapon systems; or some combination of all of the above. By some measures, 
the system is failing miserably. GAO's report that major defense acquisition programs are expected to 
overrun their budgets by $296 billion is definitely a failing grade, if you judge by systems in the field, 
you get a better grade. U.S. troops have the best equipment of any military in the world, bar none. 

What is perhaps most troubling to this committee, which has oversight as a primary responsibility, is 
that we really don't know the answer to this question. We don't have reasonable, reliable, and 
complete metrics that tell us the acquisition system is succeeding or failing. 

Hie purpose of today's hearing is to try and develop a better understanding of how we might 
develop those metrics for the future. Our witnesses today have all studied the acquisition system 
deeply, and in fact, all have decades of experience as practitioners in the acquisition field, as well as 
their academic expertise, i would note that the House-version of the acquisition reform legislation 
currently pending, known as the WASTE TKO Act, directs DOD to address the question of performance 
assessment in a significant way. The conference committee on that bill will be meeting later today, and I 
look forward to bringing some of the wisdom from today's hearing into that process. Now let me 
recognize my colleague from Texas, Mr. Conaway for his opening remarks. 
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Defense Acquisition Reform Panel 
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May 19, 2009 


Good morning, Mr. Chairman, ladies and gentlemen. I would like 
to extend a welcome on behalf of Ranking Member Conaway, as well. 
He is sorry he could not be with us today. I would like to thank the 
Chairman for allowing me to make a few introductory remarks in Rep. 
Conaway's place. 

Today's hearing is an appropriate follow-on to the Panel's two 
hearings, which focused on how DoD and GAO currently assesses 
performance on major weapons systems and service contracts. The 
purpose of this hearing is to think outside the box about how we should 


measure performance, and about what we should be measuring, and 
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less about the how we are doing it today. Our previous hearings 
revealed that current measures of performance tend to break down if 
the program baseline is unrealistic. We want to know if this problem 
can be corrected. Furthermore, are there metrics beyond cost and 
schedule performance that are of value, such as: how closely does the 
delivered system meet actual warfighter needs? Does the time of 
delivery of operational capability satisfy warfighter needs? How do we 
determine whether "optional" or "tradable" capabilities requested by 
the warfighter are affordable? 

Mr. Chairman, we have a distinguished panel of witnesses in 
front of us today. I have looked at their written testimony and I look 
forward to their testimony. There are a couple of points I would like to 
highlight. Although Mr. Dillard is not testifying today he did submit a 
written statement that I would like to briefly comment on. He states 
that, "The proliferation of autonomous fiefdoms within the Department 
continues to increase, with each being a stovepipe of oversight 
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expertise imposing unique reporting requirements, assessments and 
reviews. He goes on to state that in regards to adding additional 
acquisition workforce professionals that "These new people should not 
be housed in the pentagon, but instead where the execution of 
programs occurs." Mr. Chairman, I think this is an important point that 
I believe we need to follow closely. 

Another point or observation I would like to make is in regards to 
requirements and joint programs. We have many programs out there 
that have the word "joint" in front of them but they are joint in name 
only. I understand that the current requirements generation process is 
called the Joint Capabilities Integration System or JCIDS. It seems to me 
that in order to have a truly joint program that the requirements for 
that program must be born joint. Yet I do not believe our current 
system or as Mr. Dillard describes, current fiefdoms, foster such an 
environment. So I would be interested in hearing from our witness in 


this regard. 
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I also encourage our witnesses to share their views on existing 
laws and regulations that are particularly helpful or not helpful to the 
Department's efforts to obtain the best value and capability for our 
warfighters. We have heard from the Department in two separate 
hearings about how they currently measure performance and value on 
contracts. What we haven't heard enough about is how they should be 
measuring value. Consequently, your input will be a greatly 
appreciated. With that I will conclude and again thank my fellow 
members and you, Mr. Chairman. I look forward to the witnesses' 


testimony. 
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Mr. Chairman, members of the Defense Acquisition Panel, my name is Dave Patterson. I’m the 
Executive Director of the National Defense Business Institute at the University of Tennessee. I 
am very pleased to be here this morning to participate in the discussion of a question that has 
clearly captured the attention of the current Administration and Congress: 

How should Congress assist the Department of Defense in improving its acquisition of 
weapons and services so that it can meet the needs of the warfighter in the field while still 
being a good steward of the taxpayers’ dollars? 

Last week the Nava! Postgraduate School sponsored a Symposium with the theme of “Defense 
Acquisition in Transition.” I provided a paper for the proceedings of that symposium that 
expands on my remarks here this morning. Mr. Chairman with your approval I would submit 
that paper for the record and the Committee’s consideration in addition to my oral statement. 

The first consideration forjudging the success of an acquisition program is whether it fielded a 
weapon system, or information system or service in time to make a positive impact for the 
warfighter? A system or service fielded too late to meet the need may as well have not been 
bought. The phrase “too little, too late” can mean lost lives. 


Before we look at measures of acquisition system merit there is another consideration central to 
this discussion. When Secretary Gates made his budget announcement on April 6, 2009, 1 
believe he was speaking from frustration that was as much about what has been the persistent 
problem of having to depend on an Acquisition System that is simply not responsive to 
immediate warfighter needs as much as it was about winnowing bloated, failed or unnecessary 
programs. Implicit in that expression of frustration is a clear lack of confidence in a system that 
produces program uncertainty and instability. The most dramatic improvement metric will be 
when the senior leadership in the Administration, Congress and the Department of Defense have 
seen such improvements - results, not words - that they can say they have renewed confidence 
in the stability, predictability and effectiveness of the Defense Acquisition System. 

The Defense Acquisition Performance Assessment report contended that program stability and 
predictability were singularly and uniquely crucial to managing programs that were on cost, on 
schedule and performing. To that end in the time I have, allow me to describe two areas of 
improvement for measuring program effectiveness worthy of attention. 

First, Major Defense Acquisition Programs or MDAPs often start at Milestone B, the beginning 
of Engineering and Manufacturing Development with critical staff positions vacant. Percentage 
of critical staff positions filled at Milestone B is an easy and important metric to be observed. It 
makes little difference to implement programs to raise the level of skills of the program staff if 
they are missing in action. Programs must have full staffing at the outset in order to have a 
chance of success. 

Second, the acquisition strategy document that is to lay out how the weapon system is to be 
acquired, the initial road map, if you will, is often flawed in that it focuses more on presenting 
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the case for required capabilities and quantities than on laying out the reasoning for the 
acquisition competition methodologies. For example, both how the prime contractor participants 
in an MDAP competition will select subcontractors and how the winner of the competition will 
manage the subcontractors to gain improved efficiencies and effectiveness are generally given 
little consideration. 

Creation of the acquisition strategy document is one of, if not the most important tasks the 
government acquisition program management can undertake. The strategy should establish the 
template for all the activities that will take place throughout the source selection process, 
engineering, manufacturing and development, and follow-on production and fielding. More 
important it establishes how the program management team is thinking about the numerous 
events and activities that a program will encounter. The Defense Acquisition Executive should 
establish a common set of strategy elements that all Military Department, Service Acquisition 
Executives must include in MDAP the acquisition strategy document. Additionally, a set of 
standards or metrics by which the strategy elements can be evaluated as effective must be part of 
this process. 

When the lease proposal for the Air Force’s first attempt at procuring a replacement for the KC- 
135 aerial refueling aircraft was first introduced the Single Acquisition Management Plan I saw 
was about six pages, unsigned. Inarguabiy, this was inadequate. When criticism of the program 
began to grow, the Air Force did not have a structured, disciplined playbook to refer to and 
consequently, the defense of what they proposed was made even more difficult. 

In closing, I would be remiss if I didn’t acknowledge the progress that has been made by the 
Department in improving the acquisition system over the past four years. Though it is the 
General Accountability Office’s headline that the 96 Major Acquisition Programs have grown in 
cost by $296 billion that gets attention, those numbers belie an equally worthy, but over looked 
statistic published in the same GAO report. The average increase in unit cost of the 28 MDAP 
programs with less than five years since development start is only one percent. Compared with 
an average of 55 percent increase in acquisition unit cost of 25 programs in the group with five to 
nine years since program development start. There has been improvement that should be 
recognized. 

With that, it has been by privilege to be with you this morning and I welcome your questions. 
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Abstract 

More than three years have passed since the Defense Acquisition 
Performance Assessment (DAPA) project was completed and the results briefed 
to the study's sponsor, the Deputy Secretary of Defense. In that time, the 
Department of Defense has issued its fourteenth major change to the 
Department’s Acquisition System management guidance. Combined with a 
shortfall of experienced and skilled acquisition business professionals, the result 
is a pervasive and troubling level of uncertainty and unpredictability regarding 
defense acquisition programs. The resulting Acquisition System including 
Planning, Programming, Budgeting and Execution (PPBE), Requirements and 
the little “a" Acquisition process lacks structure and discipline. What follows is 
persistent failures to meet cost, schedule and performance objectives. This 
paper presents a case for a mandatory set of Acquisition System rules to 
address this problem. Though by no means exhaustive, the recommended rules 
fit categories in the acquisition process, the requirements process and the PPBE 
process -- referred to here simply as the “Budget Rules.” The premise of this 
paper is that the right mandatory set of rules applied to Major Defense 
Acquisition Programs would result in weapon systems and equipment critical to 
warfighter success being fielded more rapidly on cost, on schedule and 
performing as expected. 
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Introduction 

More than three years have passed since the Defense Acquisition Performance 
Assessment (DAPA) project chaired by Lieutenant General Ronald Kadish, USAF (Ret.) 
was completed in November 2006 and the results briefed to the Deputy Secretary of 
Defense, the study’s sponsor. Since that time, there have been several more attempts 
to describe the root cause of the flaws in the Department’s Acquisition System. Most 
studies cover the same ground plowed by the DAPA project and previous studies, 
dating back to the 1985 Packard Commission Report. Despite these numerous 
evaluations of the Defense Acquisition System, none have advanced the discourse 
beyond what has been clear from the beginning. There is a fundamental lack of a 
budget process and requirements discipline that leads inexorably to programs that are 
over cost, behind schedule and not performing. Additionally, there is one factor that is 
common to serious analyses of the Department of Defense (DoD) Acquisition System. 

The general discussion of reforming the Acquisition System with its many 
subsystems, procedures, and methodologies reveals that there is a basic failure to drive 
predictability regarding what the DoD can expect as a product (fielded weapon system) 
emerging from its acquisition process. In fact there is seldom any effort to make 
“predictable outcome” a program management priority. All program managers try to 
stay within budget or cost limits, meet schedule guidelines, and produce a weapon 
system or piece of equipment that performs to the level of stated requirements. But, is 
there any real certainty that the program manager’s efforts, no matter how diligent, or 
adherence to the acquisition process will produce the desired result? The case 
presented here would answer, no. The DAPA study raised the issue of the 
government’s inability to predict cost, schedule, and performance as a self-induced 
symptom of “instability” (Kadish, 2006, January). It is that instability in acquisition 
programs that defeats efforts to meet cost, schedule, and performance objectives. 

The Problem Explained 

Recent analyses of the troubles experienced in the DoD acquisition of weapons 
systems identify instability as a significant factor in program cost growth. A 2008 RAND 
Corporation study on cost growth of fixed-wing military aircraft identified the “practice of 
rotating officers through jobs every three to four years” as creating an unstable program 
management workforce (Arena, Younossi, Brancato, Blickstein & Grammich, 2008). 

This results in a management situation where experience gained in solving 
management problems is not effectively used over the term of the program and not 
available to those entering new to the program. The Aerospace Industries Association, 
in its November 2008 Special Report, U.S. Defense Acquisition: An Agenda for Positive 
Reform, raised instability as an area where the Defense Department should focus 
management attention. . 

Two elements combine to create instability in the acquisition of weapon systems. 
First, there seems to be no lasting agreement on what should be the DoD Acquisition 
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System policy directions. Since Deputy Secretary of Defense Packard issued the first 
DoD Directive 5000. 1 in 1971 , the regulations documents have been revised 
significantly about every three years: 14 times in 38 years. As Charles Cochrane (2009, 
January 1) so masterfully revealed in his presentation Acquisition Management System 
from 1971 to 2008, the DoD 5000 series documents have provided direction varying 
from 8 pages to 840 pages of recommendations, suggestions, regulations, policy, 
procedures and definitions. No single Acquisition System approach has survived for 
more than five years, while the length of time for Major Defense Acquisition Programs to 
reach full operational capability is generally three times this Acquisition System 
regulation change cycle. For the purposes of this paper, reference to the most recent 
Department of Defense Instruction 5000.02 (2008, December 8) shown in Figure I, 1 will 
be used. 


DEFENSE ACQUISITION MANAGEMENT SYSTEM 


Duration of Tim® Certain Program 
Duration of Capital funding 

J 


1 

t " 1 u 

K t f>r °s ram 

IJs Jfl i initiation) 

J 

k IOC 

FOC 

Wisteria! 

Solution 

Analysis 

Tkthnetofy Engineering and 

Develop,-, lent Manufacturing 

Development 


Production and 
Deployment 

Operations and j 
Support j 

i ttb&nrital 
\ tew&fepiwnt 
Ocdiktin 

• Post- 

PDRAy CDftA ^ 


A PK* 

< > Decis ion 

LfMP/rOT&l v ftev'i'vs 



\ Pre-Systems Acquisition /\ Systems Acquisition h Sustainment 


fy - Decision Point , - Milestone Review h / " Decision Point it PDR is not conducted before Milestone 8 

Figure 1. Defense Acquisition Management System 
(used to identify where rules described later in the paper will apply) 

Second, while the acquisition playing field is persistently changing, the workforce 
challenged with making the system successful has been reduced in numbers and 
experience. In the past, even though there were frequent modifications to the 5000 
series Department guidance, there was also a cadre of experienced acquisition 
executives in the career ranks that could adjust with a modicum of disruption. The 
United States Senate and House Armed Services Committees, in their respective 
committee reports supporting the FY 2008 National Defense Authorization Act, were 
very concerned that the numbers, years of experience and skill levels of the 
professional acquisition workforce had reached unacceptably low levels (US Congress, 
2008). Particularly troublesome was the major reduction in the acquisition workforce 
within the Department of Defense during the 1990s, the workforce on whom the 
Department counted to make sense of the constantly changing 5000 series Department 
guidance. 


1 Adapted for use in this paper from the graphic presented on page 12 of the DoD) 5000.02. 
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However well-intentioned and necessary the Department’s changes to the 5000 
series guidelines were thought to be, the consequence was instability in acquisition 
programs — an unfortunate result of a purposeful action by department management, 
instability drives uncertainty, creating an Acquisition System environment where the 
program outcome is unpredictable. When the program outcome is unpredictable, 
program risk is increased. There is a corresponding drive to reduce risk by increasing 
the cost as a premium or hedge against uncertainty. When the workforce does not 
have the experience to deal with program risk, because every program event is being 
seen for the first time, there is very little chance of maintaining cost, schedule and 
performance. The underpinning experience necessary to work through a “tried-and- 
true” process does not exist. 

These circumstances hold true for the Acquisition System as a whole, not just for 
the acquisition process — or little “a” (SECDEF, 2007, July) as it is generally understood 
within the acquisition community. The distinction between little “a” and big “A” is best 
summarized with the diagram in Figure 2. Program instability is reflective of a systemic 
problem inherent in the big “A” versus simply fixing a process problem in the little “a." 
The mythology that attends the Venn diagram with the intersecting circles is that there is 
integration among the elements of the Acquisition System. The implication is that each 
of the elements contributes to and gains from being associated with the others. The 
intended result is a successful program defined by being on cost and on schedule and 
performing as expected. The reality is more accurately represented by Figure 3, 2 in 
which the three elements exist independently of one another by virtue of the fact that 
changing regulations and vague Acquisition System direction combined with an 
inexperienced workforce allows the independence to persist (Kadish, 2006, January). 
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Figure 2, The Acquisition System 
(This includes the Planning, Programming, 
Budgeting and Execution process and the 
Requirements process in addition to the 
little “a” Acquisition process.) 



Figure 3. The Acquisition System 
(in reality, the System is not cohesive, but 
more often three independent processes 
creating program instability resulting in 
cost increases schedule slips and 


2 Both Figures 2 and 3 are adaptations of figures used in the DAPA Report, p. 4. 
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uncertain performance.) 

What results from the combination of changing acquisition regulations and a 
workforce that does not have a high enough number of acquisition professionals or the 
experience of seeing and working through a variety of program issues, is an inability to 
anticipate and prevent situations that put programs in jeopardy of failing the cost, 
schedule and performance standards. The DAPA study found that unstable programs 
did have a workforce component that contributed to the instability, and though there was 
also recognition during the subject-matter expert briefings that changing regulations and 
guidance might be troublesome in establishing stable programs, the combination of 
these two factors was not made prominent in the final report. Numerous studies have 
recommended solutions to the shortcomings of the Acquisition System, but for the most 
part, these fixes focus on the little “a" acquisition process, not on the larger systemic 
issues. 


A focus on the acquisition process (“a”) fails to address the larger contextual 
issue of the system-driven program instability. Mandating a rule set is necessary to 
establish discipline and structure. “Following the rules” helps to create an acquisition 
program where uncertainty and the resulting program instability are reduced. The need 
for acquisition program discipline was emphasized by Dr. Ashton Carter, newly 
confirmed Under Secretary of Defense for Acquisition, Technology and Logistics. 

Quoted in DefenseNews.com from his written testimony presented at his confirmation 
hearing, Dr. Carter made clear his position: “Development, procurement and 
sustainment of major weapon systems require experience with the Department of 
Defense and the defense industry, systems engineering at every stage and iron 
discipline” (Bennett, 2009, March 26). 

The following are a set of rules for defense acquisition programs that resulted 
from the DAPA panel discussions, interviews, and subject-matter expert surveys 
conducted during the DAPA project. This paper diverges from the DAPA project in that 
what the 2006 study presented as “recommendations" for consideration by the 
Department of Defense are offered here as “rules” to be followed. Additionally, the 
DAPA recommendations focused on six categories affecting the Acquisition System: 
organization, industry, workforce, requirements, budget and acquisition. However, only 
the last three categories are addressed in this paper as particularly appropriate for 
establishing rules to abide by for the acquisition of defense weapon systems. The list of 
rules is by no means exhaustive, but, rather, the list is intended to establish a 
foundation upon which additional rules may be considered, developed and applied. 
Rules that all the participants in the Acquisition System play by and are accountable to 
adds a level of transparency and predictability that can provide for stable programs. 

Requirements Rules 

For the purposes of this paper, two basic types of requirements are considered: 
customer requirements and derived requirements. Customer requirements are very 
straightforward and defined at the macro-level by approved Key Performance 
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Parameters (KPPs) 3 and non-Key Performance Parameters. 4 Derived requirements, on 
the other hand, are requirements that the customer has not specified directly as a 
requirement but that emerge or derive from the design decisions that are made 
(Brooksby, 2003). 5 

Derived requirements are not capabilities that the customer specifically has 
identified. Particularly troublesome is a subset of derived requirements that fall into the 
category of engineering changes — those changes that improve on “good enough" and 
that have a combined effect of driving up costs and missing schedule milestones. In the 
absence of rules that prevent pursuing this type of engineering change as a derived 
requirement, the guiding thought process follows this logic: “because we can, we 
should: because we should, we must; and because we must, we will no matter how 
much it costs or how long it takes.” According to a recent Under Secretary of Defense, 
Comptroller study, prepared by Monitor Company Group, L.P. and based on Selected 
Acquisition report data, engineering changes account for approximately 33% of the 
nearly $265 billion in program cost growth from 2000 to 2007 (Monitor, 2007). No 
doubt, some of the engineering changes were to correct design problems. However, 
the engineering changes that simply improve on an otherwise sufficient, specification- 
compliant design while driving up costs and impacting schedule need to be reduced or 
eliminated. 

As a result, what follows are recommended rules with appropriate rationale that 
should apply when considering the addition of both new customer requirements and 
derived requirements. 

Requirements Rule One : Weapon system requirements will be fixed prior to 
Engineering and Manufacturing Development (EMD) or achieving Milestone B phase 
(see Figure 1 on page 4). 

The prohibition of additional system requirements beyond the KPPs and the 
specific capabilities that contribute directly to them after approval for the EMD phase at 


3 “Those attributes or characteristics of a system that are considered critical or essential to the 
development of an effective military capability and those attributes that make a significant contribution 
to the characteristics of the future joint force as defined in the Capstone Concept for Joint Operations. 
KPPs must be testable to enable feedback from test and evaluation efforts to the requirements process, 
KPPs are validated by the Joint Requirement Oversight Council (JROC) for JROC Interest documents, 
and by the DOD component for Joint Integration, Joint Information, or Independent documents. 
Capability development and capability production document KPPs are included verbatim in the 
acquisition program baseline” (CJCS, 2007, May 1). Occasionally, (some would say all too often) KPPs 
cannot be achieved with the level of technology existing now or in the foreseeable future. Approval of 
this category of KPP suffers from collective bad judgment, and no rule set will be a remedy. 
Consequently, this article does not address the development of this type of requirement. 

4 Non-Key Performance Parameters are requirements that are desired by the customer but not deemed 
critical or essential. Often, these requirements represent the trade-space in programs when budget 
constraints or program execution problems demand a de-scoping of the program. 

5 Though this reference defines requirements as they apply to software development, the relevance to 
weapons system program development generally is very compelling and appropriate and, therefore, is 
used here. 
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Milestone B helps to ensure that Initial Operating Capability (IOC) will be met. Fielding 
weapon systems on schedule simply must be a program priority. By allowing 
requirements to be adopted beyond those identified prior to EMD, ensures that IOC will 
be slipped and the weapon system will riot be fielded on schedule. 

Requirements Rule Two : From the start of EMD (Milestone B, program initiation) to 
IOC, only safety-of-flight or other safety-related engineering changes will be allowed. 

The only exceptions are those design changes that can be proven to produce a three- 
to-one savings to investment while not missing schedule. 

This rule addresses the insidious nature of an ever-growing number of 
engineering changes that routinely skulk their way into systems development. 
Additionally, the rule provides a potential for incentives that produce beneficial 
engineering changes and cost savings. Though some will attempt to insert engineering 
changes using “safety” as justification, specious arguments for such justification at least 
will have increased scrutiny, prompted by the deviation from rule two. 

Requirements Rule Three : Any and all additional system requirements that are deemed 
essential following the start of EMD will be developed as unique block-up grades that 
will be introduced as blocks or variations after Full Operational Capability (FOC) has 
been certified ( see Figure 1 on page 4). 

There is a persistent need for a disciplined and structured way of incorporating 
meritorious capabilities enhancements to a weapon system while not disrupting the 
established design, cost, schedule, or performance. By following this rule, there is the 
added benefit of having some level of operational experience that can inform the 
development and insertion of weapon system improvements. 

Requirements Rule Four ; Holding to an established Initial Operational Capability as a 
time-certain for fielding the weapon system will be a Key Performance Parameter. 

Weapon systems development and fielding plans must have some consideration 
of time-to-need as integral to the requirement for the capability. This rule makes the 
time-to-need, or fielding, an essential consideration in program development and 
planning. If there is no fully understood and accepted time by which a weapon system 
must be fielded, the importance of the capability to meet a threat is called into question. 

Budget Rules 

According to the DAPA report (Kadish, 2006, January), budget instability is a 
major contributor to acquisition program instability and the failure of acquisition 
programs to meet cost expectations. Lack of funding discipline on the parts of 
Congress, the military, and the Defense Department produces acquisition programs that 
are targeted as bill payers for other funding priorities or that are under-funded because 
of poor cost estimating. 
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In his written confirmation statement submitted to the Senate Armed Forces Committee 
and reported in DefenseNews.Com, Dr. Ashton Carter emphasized the importance of 
having “stable funding” (Bennett, 2006, March 26). He considered stable funding a key 
factor in choosing whether a weapon system contract is a fixed-price type contract or 
cost-plus (2006, March 26). 

The DAPA report offers the following solution: the establishment of a funding 
account for the duration of the acquisition program from the program initiation at the 
beginning of EMD to IOC, referred to as a “Stable Program Funding Account" (see 
Figure 1 on page 4). In this article, the term “Capital Funding” is used to describe a 
stable funding account during the period from Milestone B, EMD to IOC that is tied to 
specific programs and funded by the individual Services with a fixed budgeted amount. 
Capital funding will apply initially only to MDAPs, though other acquisition programs 
could be considered. The Office of the Secretary of Defense and the Services will 
guarantee that programs identified for capital funding will not be used to pay other bills. 

Budget Rule One : All Major Defense Acquisition Programs (MDAPs) will be evaluated 
as candidates for capital funding. 

Though not all acquisition programs are suitable for a capital funding approach, 
MDAPs should at least be considered since these programs — because of their size — 
offer the most potential for reduced cost growth based on a guaranteed stable funding 
profile. 

Budget Rule Two : Capital funding programs will: 

a. Have a fixed-funding profile from Milestone B (EMD) to Initial Operating 
Capability. Capital funding programs will not be used as bill payers during that 
timeframe. 

The timeframe for capital funding allows for follow-on increases in the unit 
quantity for the acquisition program after IOC while helping to ensure that fielding the 
program is on time. Put another way, this rule helps to ensure that funding is not the 
reason for not fielding a program on time. 

b. Provide bi-annual reports to Congress on cost-schedule and performance 
progress. 

Congress’s responsibility and right for oversight of Defense spending must be 
addressed. By engaging with congressional staffs and principals to keep them informed 
of how effectively the Defense Department is spending taxpayer dollars for acquisition 
programs, the needs of Congress will be addressed. Frequent, statutorily mandated 
program reviews will provide Congress the opportunity to assess not only the program’s 
progress but also the effectiveness of capital funding. The program should be reviewed 
with Congress twice annually. This provides congressional staff and principals an early 
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understanding of developing trends. Failure to have a successful review (over cost, 
behind schedule or failing to perform) is addressed later in this paper. 

c. Have a Technology Readiness Level of at least 6 at Milestone B (EMD). 

Programs that move into EMD that do not have a Technology Readiness Level of 
6 or better are destined to experience cost escalation, schedule slips, and unpredictable 
performance. Capital funding is predicated on the fact that costs can be controlled and 
schedule can be maintained. For capital funding to be effective, all aspects of an 
acquisition program must have as much stability as possible. 

d. Be "time-certain" programs. 

Capital funding success depends on strict adherence to a fully-agreed-to 
timeframe (by the government and the contractor) from Milestone B approval for EMD to 
IOC. This provides predictability regarding what to expect in the program in general. It 
also drives the government and industry program managers to be realistic in what they 
promise for the program and in how they propose to meet the program milestones to 
stay within the timeframe for system fielding. 

e. Be cancelled if the program fails to meet established cost, schedule, and 

performance. 

If a program fails to meet any one of the cost, schedule, and performance 
objectives established at program initiation after three consecutive congressional 
reviews, the program will be cancelled; not re-baselined or re-planned — cancelled. 

When government and industry program managers as well as the military departments 
and Defense Department program executives fully understand the consequences of 
program failure, the likely result will be greater management attention. 

Acquisition Rules 

Analysis of acquisition programs over time shows that programs generally grow 
about 50% in cost (Younossi et ai. , 2006). Larger Defense programs clearly are more 
prominent when analyzing program cost growth because the amounts of money are 
very large compared to programs managed by other Federal agencies. Though it may 
seem obvious, programs that have longer timeframes for EMD also experience greater 
cost growth (Younossi et al. , 2006). Furthermore, missing from most, if not all, 
acquisition strategies is analysis that asks: “What does time, as an independent 
variable, do to the trade space defined by the minimum and optimum performance and 
cost?” 


To address the importance of time as a consideration in developing acquisition 
strategies, the Special Assistant to the Deputy Secretary of Defense asked The Monitor 
Group (2003) to look at the value of establishing time as a boundary condition or driver 
in determining the desired timeframe between Milestone B and IOC. Time should be 
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considered an independent variable, as should cost, especially when it is critical to field 
a capability in time to have a positive impact on a threat. 

Time is not the only factor that works against well-run acquisition programs. We 
have developed an acquisition-workforce culture that has adopted “flexibility” as a 
means to acquiring more capable weapon systems, other equipment, and services. 

The consequence of this culture is that there is a deliberate attempt on the part of the 
acquisition community to establish the broadest interpretation of what constitutes best 
value, desired technology, and solicitation outcome. Unfortunately, “flexibility” often 
comes at the expense of discipline and structure as a means to achieve cost, schedule, 
and performance objectives. 

Successful competitive solicitations, however, depend on discipline and structure 
in the way that the acquisition competition is managed. Competition management 
begins with development of and adherence to an acquisition management/master plan 
or strategy. That plan or strategy should inform the Request for Proposal and is the 
roadmap for the subsequent competition and program management. 

Acquisition Rule One: No MDAP will be considered for Milestone B certification without 
a comprehensive Single Acquisition Management Plan/Strategy to include at a 
minimum total system procurement quantity, explanation and rationale for the 
contracting methodology selected (i.e., prime contractor choosing subcontractors , 
leader-follower prime contractors, etc.), sustainment plan and how the Prime Contractor 
or Lead Systems Integrator will select and manage subcontractors. 

Most, if not all, programs that experience significant problems with cost, 
schedule, and performance have inadequate or flawed acquisition strategies or 
management plans. Often, the focus of the acquisition strategy is on what the weapon 
system should do, the plans of the Military Services to field the system, and the phasing 
of the number of units over time that are required. This approach, while important, does 
not comprise an acquisition strategy or management plan. The acquisition strategy 
should explain how the competition will be run; what management, technical, and cost 
elements are most important; and whether it will be a winner-take-all (and why that is 
the preferred choice), split-buy, leader/follower strategy, or some combination of each. 
These considerations in an acquisition strategy are important and will drive necessary 
program decisions in the follow-on program management. 

Management and acquisition strategies should consider what must be fielded 
and when and how block upgrades will be completed, managed, and integrated after full 
operating capability is achieved. The acquisition strategy must describe how the 
winning contractor will manage subcontractor content. An annex to the acquisition 
strategy must be how the weapon system competition will be financed, and 
consideration must be given to any subsystem's commercial value in terms of design 
buy-back and production rights. In the past, the Department has either retained all of 
the design rights or retained none of the rights. Retaining substantial design rights 
while keeping open the opportunity for the contractor to benefit from any commercial 
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markets that might emerge makes competing for the Department’s business more 
appealing. 

Acquisition Rule Two : All MDAPs will be evaluated as "time-certain’’ programs, where 
the timeframe between Milestone B and IOC (see Figure 1 on page 4) will be 
established with a thorough analysis, using Time as an Independent Variable (TAIV). 
Additionally, the criteria that describe what must be accomplished in the EMD phase of 
the program cannot significantly change. 

When TAIV is applied to the development of an acquisition program, the 
importance of time in developing and defining the technology, as well as its design and 
production factors, are given prominence in the analysis of cost, schedule, and 
achieving the desired performance. Time-certain in this instance is not synonymous 
with schedule. Schedule is the sequential distribution of program events that, on 
completion, have a timeframe associated with them. We measure schedule with 
milestones accomplished. TAIV, on the other hand, is the analytic construct that 
identifies which out of a given list of performance capabilities are of marginal value 
when considering the amount of time necessary for a capability to be developed, 
incorporated into the weapon system, and fielded. The time-certain period is 
established with the results of the TAIV analysis. Schedule is, then, the sequence of 
events or program milestones that fit within the time-certain period. Though a recent 
Government Accountability report (GAO, 2009) points out that the DAPA report (Kadish, 
2006, January, p.49) recommended that schedule be a Key Performance Parameter, 
this rule departs from DAPA in that the time between Milestone B and IOC be a time- 
certain period and that specific length of time be a Key Performance Parameter. 
Urgency for fielding a particular desired capability, then, has a context that can be used 
to describe what needs to be fielded or deployed and when. 

Acquisition Rule Three: Aircraft programs will take no longer than five years from 
Milestone B (EMD) to Initial Operational Capability, again using TAIV as an analytic tool 
to validate the optimum timeframe. 

Successful aircraft programs have been fielded in five years or less. The fielding 
of both the F-15 and F-16 were achieved in approximately five years, with the F-15E 
(Woods, 2008) fielded in approximately five-and-one-half years. Had management and 
budget attention been constant and sufficient, the C-17 cargo aircraft could have 
achieved IOC in five years. But after several false starts, it took almost 10 years. The 
complexity of the aircraft’s technology demands is clearly important, but other factors 
seem to play roles as well. The EA-18G is planned for five years and nine months from 
Milestone B to IOC while the F/A-18E/F was planned for nine years and four months. It 
is true, however, that the EA-18G is basically an F/A-18F airframe integrated with an 
Improved Capability III, Airborne Electronic Attack (AEA) avionics suite (employed on 
the EA-6B) and should take less time to field. The accelerated development schedule 
(over its F/A-18E/F predecessor) probably can be attributed to the coupling of that 
proven, in-production airframe with an existing AEA technology. 
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The B-2 took 18 years from Milestone B to IOC for a variety of reasons, only 
some of which had to do with available technology. Budget and congressional interest 
played big roles in the length of time that it took for the B-2 to reach IOC. At 14 years 
and four months, the F-22 has taken the longest of any of the fighters to reach IOC. If 
the rules are followed that require capital funding and not being certified for EMD 
without achieving a TRL of 6, it is not a stretch for a well-managed program with stable 
funding that follows all of the rules described in this paper to reach IOC in five years. 
However, when the program becomes a bill payer for other Service needs or derived 
requirements are inserted before or during the EMD phase, five years will, of course, be 
a difficult achievement. The criteria that describe what must be accomplished in the 
EMD phase of the program cannot significantly change. 

Acquisition Rule Four. Ship-building programs will take no longer than seven years 
from Milestone B (EMD) to Initial Operational Capability, again using TAIV as an 
analytic tool to validate the optimum timeframe. 

Currently, the average time from Milestone B to IOC for US Navy ships entering 
the fleet is eight years, nine-and-one-half months (Costello, 2008). Size and 
complexity, however, do not seem to be what determines the length of time to get 
combat ships into the fleet. The range is from CVN21 (Gerald R. Ford Class modern 
aircraft carrier) taking 12-and-one-half years to strategic sealift ships taking five years, 
nine months. But, again, complexity or size does not seem to be the driving factor since 
a Supply Class Fast Combat Support Ship (AOE6) took over eight-and-one-half years to 
go from Milestone B to IOC while the aircraft carrier CVN74, USS John C. Stennis, took 
a little over eight years, four months to achieve IOC. An LPD 17 San Antonio Class 
amphibious docking ship took 1 1 years and one month to reach IOC while the Arleigh 
Burke Class (DDG 51) destroyer took nine years, three months to go from Milestone B 
to IOC. Arguably, to establish seven years as the time-certain for naval shipbuilding 
programs from program initiation at Milestone B to IOC will be a challenge — but a 
challenge that can be met if the time-certain constraint is one that both the contractor 
and the Department understand and capture in their Integrated Master Plan and 
Integrated Master Schedule. Also, the criteria that describe what must be accomplished 
in the EMD phase of the program cannot significantly change. 

Acquisition Rule Five: Requests for Proposals (RFP) will include a competition element 
that asks how the competitors plan to select, manage, and evaluate their 
subcontractors. Subcontractor management will be an element of the Contractor 
Performance Assessment report and considered in determining award and incentive 
fees. 


With regard to the issue of subcontractor management raised in Rule Four, how 
the winning prime contractor intends to select and manage its subcontractors and 
suppliers will be a prominent competition element in the Request for Proposal (RFP). 
The purpose of this rule is to discourage potential prime contractors from arbitrarily, and 
as the default position, choosing sister divisions as subcontractors. If a competitor must 
explain the rationale for selecting subcontractors' contributions and their cost and 
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design advantages compared to sister divisions or other alternatives, sister divisions 
may not be as appealing of a choice as a program subcontractor. Knowing that the plan 
for selecting and managing subcontractors will be weighted in the management section 
of the RFP will provide more incentive to the potential prime contractor to give very 
careful consideration to subcontractor selection. Profit-on-profit should become more of 
a competitive liability. 

Acquisition Rule Six: No MDAP will be considered for Milestone B certification without a 
Test and Evaluation Master Plan that has been agreed to and approved. 

All too often, the test and evaluation process results in new requirements that 
exceed contract specifications. Ensuring that a fully agreed-to and approved Test and 
Evaluation Master Plan that clearly bounds the limits of what can be tested, including 
metrics for success that all understand, is essential. This will go a long way to 
precluding testing the driving engineering changes and requirements that exceed the 
contract defined design. 

Acquisition Rule Seven: Where the competitors offerings are comparable and the 
competition will allow, competitors for EMD will submit cost-model data and Most 
Probable Cost will be determined prior to final Request for Proposal release and shared 
with competitors. Most Probable Cost will be contract cost. Competitions will be based 
on technical and management risk. 

A long-held view in the defense industry is that any program vice president who 
loses a cost competition by not having the lowest cost is fired. What exactly drives the 
industry to hold this point of view? If you don't count their years of experience, a 
winning contractor believes that there is a better than 80% probability that the contract 
specifications that were bid will be changing as the ink is drying on the contract. The 
winning contractor can then charge full price on the updated program specifications, 
within the cost and pricing guidelines, and make up for any risk accepted in the original 
winning proposal. 

This approach to an acquisition program is most often prevalent when the 
contract is a cost-plus arrangement, though fixed-price contracts experience the same 
type of expanding-contract cost growth with the emergence of derived requirements and 
engineering changes. The problem that occurs with fixed-price contracts that have 
engineering changes or derived requirements is that unless the contract is amended, 
the cost of the changes often turns up as claims against the government. Cost-plus 
contracts, on the other hand, only have the added costs show up as the “plus” in cost- 
plus. 


The excuse often heard when costs rise is that the Department and the defense 
contractors do not have good cost estimates. This assertion does not generally prove 
to be true. When competing contractors reveal the output of their cost models and 
compare them with the Department’s estimates, there is often very little difference. As a 
general rule, then, all the participants in acquisition competitions have a very good 
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estimate of what the costs will be. Why then are competitions based on cost when 
everyone knows what the most likely cost will be? Cost should be taken off the table 
and the competition should be about which competitor has the better solution for 
management and technical risk, with subcontractor selection and management being 
prominent in that evaluation. Most Probable Cost, or the cost that the competitors and 
government models agree is the cost, should become the contract cost. 

Acquisition Rule Eight: Competitions will be based on the motto: “the design you bid is 
the design you build. " 

A number of activities take place while the ink is drying on the contract. Not the 
least of these is that the government program manager and executive are saying to the 
winning contractor, “We know what we said we wanted, and what you proposed, but we 
have a few changes to the requirements we'd like you to adopt.” To which the winning 
contractor readily replies, “Not a problem; just a few design changes, another year on 
the program, and an increase in cost." If both the winning contractor and the 
government program manager fully understand and believe that the design that was bid 
is the design that will be built, then their behavior will change to follow the rule. 


Conclusion 

The Department of Defense is now in a budget environment where it is directly 
competing with a formidable domestic agenda that will not be denied. In his January 
2009 article A Balanced Strategy: Reprogramming The Pentagon For A New Age, 
published in Foreign Affairs, Secretary of Defense Robert Gates stated, 

In recent years, these platforms have grown evermore baroque, have 
become ever more costly, are taking longer to build, and are being fielded 
in ever-dwindling quantities. Given that resources are not unlimited, the 
dynamic of exchanging numbers for capability is perhaps reaching a point 
of diminishing returns. A given ship or aircraft, no matter how capable or 
well equipped, can be in only one place at one time. (Gates, 2009) 

If Secretary Gates' message is going to be taken to heart by those charged with 
acquiring the “platforms” and those responsible for producing them, then far greater 
attention must be given to using the defense budget wisely, efficiently and effectively. 
Programs simply must be managed to cost, schedule and performance. A mandated 
set of rules that drive discipline into the Acquisition System is one answer. 

This paper describes a few such rules that are worthy of implementing. They are 
by no means inclusive of all the rules that should be considered and established. 
Additionally, it should be clearly understood that for behaviors to change, all of the rules 
must be followed since no single rule or group of rules stands alone. For example, 
without a time-certain program, the discipline for capital funding will not be present and 
planning for funding over a well-defined time period will not be possible. The rules are 
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interrelated, and these rules are necessary in order to re-establish an acquisition culture 
that is disciplined with a clear understanding of how to bring predictability and stability to 
the Department of Defense Acquisition System. 

The institution of rules that are clearly communicated and consistent must be 
enforced to reinvigorate and support the acquisition workforce’s enthusiasm for meeting 
cost, schedule, and performance as well as establishing discipline and structure in the 
Acquisition System. The Department of Defense establishes and follows checklists for 
any number of activities from flying airplanes to mailing packages. Rules are just 
another form of a checklist. With 91 major Defense Acquisition Programs with a 
combined value of over $1.6 trillion currently being managed, the result will be getting 
better weapon systems into the hands of the warfighter in time to make a difference on 
the battlefield. 
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as developing legislative strategies, and developing and implementing DoD financial policy, financial 
management systems, and business modernization programs. In June 2005 Mr. Patterson was 
appointed to lead the Defense Acquisition Performance Assessment Project, a comprehensive 
evaluation of every aspect of the Defense Department acquisition system and decision making 
processes. 


From August 2003 to June 2005, Mr. Patterson held duties as The Special Assistant to the Deputy 
Secretary of Defense. In the capacity as Special Assistant, Mr. Patterson was responsible for 
managing the Deputy Secretary of Defense’s personal staff as well as providing direction and advice 
to the Office of the Secretary of Defense Staff on a wide range of national security operations and 
policy subjects. He contributed to the Department of Defense support to the United States' mission 
to establish free and economically successful societies and governments in Iraq and Afghanistan. 
Additionally, Mr. Patterson supported the Deputy Secretary in the areas of military commissions for 
detainees in the Global War on Terrorism and major defense acquisition programs! 

Before returning to government service, Mr. Patterson was a founding and managing partner at 
Bucher, Hutchins, Kohler and Patterson, Inc. where he led the firm’s commercial consulting practice 
developing management strategies for acquiring new business. From 1999 to 2001, he was the 
Vice President and Site Manager for Steven Myers and Associates’ support to Lockheed Martin 
Corporation's winning Joint Strike Fighter competitive proposal preparation. 

Between 1993 and 1999, Mr. Patterson held a variety of responsible, executive positions at 
McDonnell Douglas Corporation (later The Boeing Company) beginning as the Senior Manager for 
Market Research and Analysis on the C-17 military air cargo aircraft and later as Director, 
International Business Development. He was responsible for developing and executing the business 
capture strategy that won U.S. Government Defense Acquisition Board approval to procure 80 
additional C-17s completing the first contract for 120 aircraft. Mr. Patterson led the Boeing business 
development team that launched the initiative to introduce a commercial version of the C-17; the 6C- 
17. , 

Mr, Patterson served in the Air Force from 1970 to 1993 retiring in the rank of colonel. During that 
time, he held responsible leadership and management positions with assignments at the air wing 
level as a C-5A aircraft commander and Deputy Operations Group Commander, at major command 
headquarters, Headquarters, U.S. Air Force, the Office of the Chairman, Joint Chiefs of Staff and the 
Office of the Secretary of Defense, Inspector General. In 1986, Mr. Patterson was the Air Force 
Fellow at the American Enterprise Institute. He served in Vietnam flying Q2As as forward air 
controller. 
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DISCLOSURE FORM FOR WITNESSES 
CONCERNING FEDERAL CONTRACT AND GRANT INFORMATION 


INSTRUCTION TO WITNESSES: Rule 11, clause 2(g)(4), of the Rules of the U.S. 
House of Representatives for the 11 l lh Congress requires nongovernmental witnesses 
appearing before House committees to include in their written statements a curriculum 
vitae and a disclosure of the amount and source of any federal contracts or grants 
(including subcontracts and subgrants) received during the current and two previous 
fiscal years either by the witness or by an entity represented by the witness. This form is 
intended to assist witnesses appearing before the House Armed Services Committee in 
complying with the House rule. 


Witness name: Jack David Patterson 
Capacity in which appearing: (check one) 
Individual 




X Representative 

If appearing in a representative capacity, name of the company, association or other 
entity being represented: National Defense Business Institute, University of Tennessee 
FISCAL YEAR 2009 


federal grant(s) / 
contracts 

federal agency 

dollar value 

subject(s) of contract or 
grant 

F A7 0 1 4-06-D-00 1 9-009 

U.S. Air Force 

$ 890,809 

Major Defense Acquisition Program 
Indirect Cost Study 

FA7014-06-D-00 19-009 

U.S. Air Force 

$ 601,597 

Defense Acquisition Program 

System Gap Analysis 

FA7014-06-D-001 9-009 

U.S. Air Force 

$1,201,806 

Program & Process Controls 


















FISCAL YEAR 2008 


federal grant(s) / 
contracts 

federal agency 

dollar value 

subject(s) of contract or 
grant 

FA7014-06-D-001 9-008 

U.S. Air Force 

$ 1,151,082 

Chief Management Officer Study 

FA7014-06-D-00 19-008 

U.S. Air Force 

$ 725,000 

Major Defense Acquisition Program 
indirect Cost Study 






















FISCAL YEAR 2007 
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Federal grant(s) / 
contracts 

federal agency 

dollar value 

subject(s) of contract or 
grant 

None 





























Federal Contract Information: If you or the entity you represent before the Committee 
on Armed Services has contracts (including subcontracts) with the federal government, 
please provide the following information: 

Number of contracts (including subcontracts) with the federal government: 

Current fiscal year (2009): 1 ; 

Fiscal year 2008: 1 ; 

Fiscal year 2007: 0 . 

Federal agencies with which federal contracts are held: 

Current fiscal year (2009): United States Air Force : 

Fiscal year 2008: United States Air Force ; 

Fiscal year 2007: NA . 

List of subjects of federal contract(s) (for example, ship construction, aircraft parts 
manufacturing, software design, force structure consultant, architecture & engineering 
services, etc.): 

Current fiscal year (2009); Acquisition - Research, analysis, training & tool 
development ; 

Fiscal year 2008: Acquisition - Research, analysis, training & tool development ; 
Fiscal year 2007: NA . 

Aggregate dollar value of federal contracts held: 

Current fiscal year (2009): $2.694.212 : 

Fiscal year 2008: $1.876.082 : 

Fiscal year 2007: 0 
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Federal Grant Information: If you or the entity you represent before the Committee on 
Armed Services has grants (including subgrants) with the federal government, please 
provide the following information: 

Number of grants (including subgrants) with the federal government: 

Current fiscal year (2009 None . 

Fiscal year 2008 None . 

Fiscal year 2007: None . 

Federal agencies with which federal grants are held: 

Current fiscal year (2009):_ ; 

Fiscal year 2008: ; 

Fiscal year 2007 . 

List of subjects of federal grants(s) (for example, materials research, sociological study, 
software design, etc.): 

Current fiscal year (2009): ; 

Fiscal year 2008: ; 

Fiscal year 2007: . 

Aggregate dollar value of federal grants held: 

Current fiscal year (2009): ; 

Fiscal year 2008: ; 

Fiscal year 2007: . 
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Chairman Andrews and Members of the Panel: 

My name is David Fitch. 1 have the pleasure of serving as a member of the 
senior leadership team at the Defense Acquisition University (DAU). I have held 
various leadership positions at the University, including nearly seven years as the 
Dean, Defense Systems Management College — School of Program Managers. 
Currently, I am the Director of the university’s Leadership Learning Center of 
Excellence. I am a retired Navy Captain and I served in acquisition or acquisition 
related positions for approximately 18 years of my 30-year active duty career in 
the Navy. 1 held leadership positions including squadron command, command of 
a Navy laboratory, Deputy to the Assistant Commander of the Naval Air Systems 
Command, and five years as the major program manager for a successful 
international and joint Major Defense Acquisition Program (ACAT ID). After 
retirement from the Navy and before returning to government service. I worked in 
the defense industry for three years. 

I want to thank you for the opportunity to appear before the panel and to 
participate in today's discussion. I will address the general subject of acquisition 
performance metrics, and your specific questions about how to increase the 
realism of program baselines, making trades between affordability and 
performance, and how to assess the value of the systems that are delivered to our 
warfighters. Please recognize that these are my opinions based on over thirty 
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years in the business and do not necessarily reflect the views of the Defense 
Acquisition University or, the Department of Defense, or the Administration. 

THE IMPORTANCE OF STRATEGIC CHOICE 

Measurement must encompass both “strategic” and “tactical” elements of 
acquisition. As emphasized in a recent Defense Science Board Report (Creating a 
DoD Strategic Acquisition Platform), what we call tactical acquisition — the 
management, execution and oversight of acquisition programs — is moot if we 
aren’t spending taxpayer dollars to buy the right capabilities — strategic choice. 

With respect to strategic choice, it is as important to decide what we won’t 
buy, as well as what we will buy. The decision on how to allocate research & 
development and procurement dollars is a strategic issue. I believe one of the root 
causes of funding instability is “too many programs chasing too few dollars” — 
this is a fundamental cause of overly optimistic cost estimates. The recently 
implemented Material Development Decision (MDD) process will bring the right 
players together. This will also increase collaboration and integration of the three 
major acquisition support systems known as; I) requirements, 2) resources, and 3) 
acquisition. This should produce better informed and disciplined investment 
decisions. The MDD process has the potential to change the DOD culture and, in 
the future, to resource programs at higher confidence levels to lower programmatic 
risk. 
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Improving the requirements process is another high potential initiative 
focused on addressing systemic acquisition issues. Having a formal requirements, 
capabilities-focused definition process is not unique to DOD and we can learn 
important lessons by benchmarking best practices from industry. If you compare 
the DoD acquisition system with a commercial market example, such as the 
development of electronic games, there are marked similarities, as well as 
differences in practice. Notably, the year long process to get games on shelves for 
the December holiday season starts with a precise clarity of what will be 
developed — by when — and includes a corporate commitment to provide the 
resources required for the project. That level of clarity is the result of intense 
interaction between the people that define the capabilities of the game and the 
people that will develop and test the software before mass production starts. 

We have recently deployed an initiative to improve the acquisition process 
by training of members of the Requirements Community on the fundamentals of 
acquisition. This initiative, supported by Congress, provides training to ensure 
requirements writers have a sufficient understanding of critical elements of 
acquisition, such as systems engineering and testing. The intent is straightforward 
and simple: to improve collaboration between the Department’s acquisition and 
requirements community throughout the acquisition life cycle to better identify 
cost and performance trades at the right decision points to enhance opportunities 
for improved acquisition outcomes. 
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TACTICAL ACQUISITION METRICS 

The most effective tools and templates incorporate metrics — both 
quantitative and qualitative. The question was raised, "Are there metrics beyond 
cost and schedule performance that are of value?" Yes, and an ongoing example is 
the "Probability of Program Success" (POPS) metrics that are currently being 
deployed across the Services and other federal agencies such as the Department of 
Homeland Security. 

The objective of POPS is to identify a system of program metrics to alert 
senior leaders to situations that might require their attention and intervention. 
Starting with a blank sheet of paper, a group of DAU faculty, experienced 
program managers and other functional experts, asked themselves a series of 
questions'. What conditions facilitate the success of programs? What metrics are 
leading indicators of derailment? Which of these elements are within the control 
or influence of the program manager and which aren’t? The resulting tool, POPS, 
is a structured process and display that describes and assesses key elements of 
planning, resourcing, execution and external influences that promote or negatively 
impact program success. 

This initiative is still evolving and is being actively used within the Army, 
Air Force, Navy, Marine Corps, and Coast Guard. The tool has been incorporated 
into DAU program manager training. Metrics in and of themselves do not produce 
success. However, when timely, accurate, and transparent metrics are integrated 
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into the management and oversight processes, better decisions and timely risk 
mitigation can be achieved. 

Among the information that goes into POPS are earned value, integrated 
master schedule, and technical performance metrics. These represent the key 
building blocks of system capability - cost, schedule and performance. Another 
important "Probability of Program Success" metric is the adequacy of personnel 
resources — numbers and competencies — in industry and government. A highly 
qualified and appropriately sized workforce is vital to achieving successful 
program outcomes. 

In addition to teaching POPS in program manager courses, we also use 
them when we are providing performance support to acquisition organizations in 
helping them solve problems. 

IMPROVING ACQUISITION PROGRAM BASELINES 

The question has been raised, "Can we ensure improved, realistic 
baselines?" We believe the answer is yes and there are several initiatives ongoing 
in the Department now. These include increased emphasis by the Department on 
assessing technology readiness and retiring risk early before starting major 
systems development. These will, in my opinion, result in better cost estimates, 
better acquisition strategies, and more realistic schedules — in short, better 
acquisition program baselines. In addition to retiring technical risk and 
producing realistic acquisition program baselines, the process of competitive 
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prototyping included in the new DoD 5000.02 and pending legislation increases 
the opportunity to appropriately address affordability and capability trades. 

The competitive prototyping process allows program teams to better define 
technology maturity, risk, cost, and other programmatic challenges earlier. 

Bottom line - this allows the government to make better decisions with actual 
performance data of competing industry teams before making a down select for 
engineering and manufacturing development. No matter how thoughtfully we 
plan or discipline the execution of source selections, a paper-only selection 
process is never as good as hard data from competing contractors. This produces 
actual results - not just promises. Competitive prototyping requires industry to 
put sufficient talent on programs or they reduce their chance of being the winner. 
Program performance and success is in doing, not just paper proposals. 

Another important change is related to conducting preliminary design 
reviews (PDRs) prior to Milestone B. The intent of this change, like competitive 
prototyping, is to give the government and industry much greater insight into 
derived requirements that may drive cost and schedule. Obtaining this knowledge 
sooner will result in better cost and schedule estimates for engineering and 
manufacturing development and more realistic acquisition program baselines. 

DELIVERING OPERATIONAL CAPABILITIES 

The ultimate assessment of whether we have delivered value and the 

needed capability to the warfighter is feedback from the field. 
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Field feedback can come in different forms. The picture above is a great 
example of the value of the products delivered and the appreciation of the 
Soldiers, Sailors, Airmen, and Marines who use these products. 

Before equipment is fielded, it undergoes rigorous levels of developmental 
and operational testing. Testing, whether at the component or system level, is the 
true indicator of a system's progress towards delivering the intended operational 
capability. The new DOD 5000.02 has increased the emphasis on oversight and 
analysis of developmental testing, to include earlier developmental testing and 
technology maturation. 

CONCLUSION 

Mr. Chairman, thank you for the opportunity to participate in this important 


discussion. 
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Mr, David P. Fitch 

Defense Acquisition University 


Mr. Fitch enlisted in the Navy in 1966. He was commissioned in 
1968 after graduation from San Jose State College and Aviation 
Officer Candidate School in Pensacola, Florida. In 1989, after 
fixed and rotary wine flight training, he was designated a Naval 
Aviator and a Distinguished Naval Graduate. He retired from the 
Navy in 1998 as a Captain following a career that included three 
operational and major acquisition command assignments. 

Mr. Fitch’s acquisition career began at the Naval Air Systems 
Command (NAVAIR) as the Assistant Project Manager for 
Systems Engineering (ARM, S&E) for H-3 helicopters, including 
the Presidential fleet of VH-3D helicopters. He managed the 
Navy’s first helicopter Service Life Extension Program and 
installation of defensive counter measures on the VH-3D. 
Following tours on the OPNAV staff as a Requirements Officer 
and as Commanding Officer of an aviation training squadron (VT- 
2), Mr. Fitch returned to NAVAIR as the ARM, S&E for three 
models of the SH-6Q helicopter when these programs were 
undergoing operational testing and transition into production. 

Next, Mr, Fitch was assigned as Commanding Officer, Naval 
Coastal Systems Center (NCSC) in Panama City, Florida, which 
was responsible for research, development and support of special 
operations, amphibious forces, and mine countermeasure units in 
the Persian Gulf during the Gulf War. As the CO of NCSC, he 
ted a major reorganization of the center. Subsequently, Mr. Fitch 
served as the Deputy to the Assistant Commander at NAVAIR. 

Mr. Fitch culminated his tour in the Navy as the major program 
manager for the international and joint Multifunctional information 
Distribution System (MIDS) — a 1997 recipient of both the David 
Packard Excellence in Acquisition Award and the Department of 
Defense Value Engineering Award. 

In 2001 , after nearly three years in the defense industry, Mr. Fitch 
joined the faculty of the Defense Acquisition University, teaching 
and consulting acquisition executives in program management 
and other acquisition disciplines. In 2006, he led a major 
independent study of the Coast Guard Deepwater program. In 
July 2008, he assumed the position of Director, AT&L Leadership 
Learning Center of Excellence after nearly seven years as the 
Dean of the Defense Systems Management College. 


Mr. Fitch holds degrees in Business and Industrial Management 
from San Jose State College and an MS in Education from the 
University of Southern California. He is a graduate “With Highest 
Distinction” from the Naval War College, Newport, Rhode Island. 
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Testimony for the House Committee on Armed Services 
- Panel on Defense Acquisition Reform - 
May 19, 2009 

Statement of Dr. Daniel A. Nussbaum, Naval Postgraduate School 

Chairman Andrews, distinguished members of the panel, I would like to thank 
you for this opportunity to discuss my thoughts on how to improve the acquisition and 
cost estimating processes in the Department of Defense. These ideas are mine and do not 
necessarily reflect the views of the Navy or the DoD. 

1 am Daniel A. Nussbaum, a member of the faculty at the Naval Postgraduate 
School, Graduate School of Operational & Information Sciences, in Monterey, 

California. I’ve spent the last thirty years mainly doing and more recently teaching and 
researching in the defense acquisition management system, with a focus on cost 
estimating. I was a previous Director of the Naval Center for Cost Analysis and past 
President of the Society of Cost Estimating and Analysis. In my current capacity, I 
educate mid-career military officers and Department of Defense civilians, as well as 
graduate students from many allied nations, in cost estimating, and I conduct research in 
cost estimating and the return on investment that the insertions of technologies offer. 


All my experiences in cost estimating confirm that three things are necessary for 


sound cost estimating. They are 1) Acceptance of the underlying uncertainties in 
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predicting the future, 2) Accurate and plentiful historical data, and 3) Professionally 
trained and certified personnel. 

First I want to discuss Uncertainty 

There is uncertainty in all estimates, and this uncertainty is intrinsic to the 
professional practices in the Cost estimating profession. Uncertainty derives from several 
sources, with a major one being that we are usually designing (or building or operating) 
something that is substantively different from what we did before. The difference may be 
in the product itself (e.g., the technologies used in the system), or in the economic 
conditions (e.g., labor rates, or overhead rates, or material costs), or in the programmatic 
environment (e.g., how many we buy at a time or in total, and the manufacturing 
processes and technologies used). 

An estimate reflects our knowledge at a point in time, when we "freeze the problem" 
and base the costs on the configuration and programmatics as they are understood at that 
moment. From that baseline, many things can change that can also change the cost 
estimate, including: 

• Changes in labor rates, overhead rates, or schedule, 

• Changes, including enhancements, to capability or quantity. While these can be 
controlled by mechanisms such as the Configuration Control Board (CCB) 
initiative. The CCB is not intended to tie the hands of the customer who may 
very well need these changes. For example, we would not peremptorily reject 
enhanced Inteligence, Surveillance and Reconnaissance (ISR) capabilities to 


1 



68 


Global Hawk, to support the kinds of missions we're engaged in now, solely for 
reasons of increased cost. 

• There are cost changes internal to the program when a particular technical 
solution to a problem does not work as planned, and an alternative technical 
solution becomes necessary. For example, aluminum may have been assumed to 
be the right material for some portion of an aircraft structure, with man-hours per 
pound of aluminum used as the basis of estimate. If subsequent testing or 
analysis shows aluminum to be inadequate, some redesign will almost surely be 
necessary. Redesign takes time and engineering labor) to accommodate the use of 
alternative materials. And, alternative materials, like titanium, cost more for 
materials as well as for fabrication. 

Second, let us look at the impact of Historical Data 

A hallmark, and necessary characteristic, of a sound cost estimate is that it is 
based on historical program performance (what we call “actuals”) from similar or related 
on-going or past programs. 

While the historical record is necessary, it is not sufficient, because we must take 
into account those processes and technologies that came into being after the dates of the 
historical data. 

Historical data is variable. Not every aircraft costs the same as every other 
aircraft. Not every ship costs the same as every other ship. This variability is at the root 
of the “less-than-perfect fit” that characterizes our cost estimating models. The 
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measurement of these “less-than-perfect fits’’ is accomplished through statistical 
constructs like “standard error of the estimate” and “confidence intervals”. 

The general point is that we assume that the patterns of the past will repeat in the 
future. These patterns are almost always statistically grounded patterns, modeled with 
the powerful and subtle techniques known collectively as “regression”. Further, we know 
of no alternative approach to using the past as a guide to the future, if we want a scientific 
(i.e., reproducible and auditable) approach. 

Grounding cost estimates in historical program performance suggests that the cost 
estimating profession subscribes to the idea that “the Past is Prologue”. That is correct. 
To the charge that such an approach is deficient in that it presumes that all the 
experiences of the past - the good and the bad - will be repeated, we in the Cost 
Estimating community answer that the future will not replicate the old mis-forecasts and 
inaccurate assumptions; rather, the future will bring with it its own, new, mis-forecasts 
and inaccurate assumptions. Of course, the same is true of the successes and correct 
assumptions of the past. That is, the historical successes and correct assumptions will not 
be exactly replicated; rather, new successes and correct assumptions will be achieved. 

By bringing statistics and probability into the arena, we accomplish two things: 
we can take advantage of the full power of these powerful, subtle, mathematical 
disciplines; and simultaneously, we have introduced the language, the methods, and the 
results of these mathematical disciplines into what are invariably public discussions. In 
this sense, we are like meteorologists, usually right in general and occasionally not right 
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in particular. It is a continuing challenge for the professional cost estimator to provide a 
clear explanation of the basis of a cost estimate to those who use those cost estimates. 

And thirdly, we need to talk about the key role of Professionally Trained and 
Certified Personnel. 

Not all estimating is done by government employees. There are three cost 
estimating sub-communities: government in house estimators; employees of the large 
vendors who also design, develop and build what we buy (e.g., Boeing, Northrup 
Grumman, Lockheed Martin); and support contractors (or, consultants) that specialize in 
(or have divisions that specialize in) supporting the cost estimating needs of the 
government. Surely, we need to increase the capacity and quality, that is, the numbers 
and training, of the government estimators, but so do we need to enhance the 
professionalism of the other two communities. 

Currently, there are no undergraduate curricula in cost estimating. The 
undergraduate classes that are closest to cost estimating are those in financial economics, 
which introduce the concept of net present value, but do not address the source or basis of 
the underlying cost estimates. 

There are four educational institutions that 1 am aware of that teach at least one 
course in cost estimating. These are 

• The Naval Postgraduate School (NPS, http://www.nps.edu/), located in Monterey, 

California, where 1 currently teach and do research in cost estimation. 


4 




71 


• The Air Force Institute of Technology (AFIT, http://www.afit.edu/), located on 
Wright Patterson Air Force Base, in Ohio. 

• The Defense Acquisition University (DAU, http://www.dau.mil/), located in 
diverse locations, with the DAU Capital campus on Ft. Belvoir, in Virginia. 

• Massachusetts Institute of Technology (MIT, http://web.mit.edu/) in Cambridge, 
MA., which offers one elective course in cost estimating within its engineering 
curriculum. 

DAU provides instruction that is required for certification in cost estimating, in the 
Business, Cost Estimating and Financial Management (BCEFM) Career Field, under the 
Defense Acquisition Workforce Improvement Act (DAWIA). The recent separation of 
the BCEFM career field into two separate CE and BFM tracks is a very welcome 
development (cf https://acc.dau.mil/CommunityBrowser.aspx?id=277653&lang=en-US 
for background) and should be supported. DAU is industriously tailoring its Cost 
Estimating courses to accommodate the separate CE track, and this too is a good 
evolution which should be supported. DAU support is largely limited to military and 
DoD employees. 

The Society of Cost Estimating and Analysis (SCEA, http://www.sceaonline.net/) 
membership includes approximately one-third of all cost estimators supporting DoD. 
SCEA is a central and indispensable player in the training, initial certification and 
periodic recertification of cost estimators. Specifically, SCEA has collected a body of 
cost estimating knowledge that it provides to members of the Cost Estimating 
community, provides training in cost estimating, has developed and offers an 
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examination and experienced-based certification program that " ...promotes competency 
recognition based on preparation, assessment, and sustainment... it provides a 
professional credential that sets the standard for the entire costing estimating and 
analysis community. It provides the foundation for professional cost careers and offers 
employers and individuals a means of distinguishing and achieving excellence. 

Ultimately, certification offers a stamp of approval of an individual's mastery of the basic 
and intermediate cost knowledge and consequently strengthens the individual's and their 
organization's ability to produce quality cost estimates arid analyses 

Speaking from my experiences at the Naval Center for Cost Analysis, I can say that 
the best way to find, recruit, train, develop, and retain the next generation of cost 
estimating leaders is to have and support a robust Cost Estimating Intern Program. Look 
at the current leadership of Navy cost estimating, and you will be amazed at the number 
who began their careers in a Cost Estimating Intern Program. 

A Cost Estimating Intern Program combines classroom training with on-the-job 
experience and enables DoD to cultivate skilled estimators with hands-on experience in 
the state-of-the-art technologies of defense weapons, platforms and support systems, the 
DoD planning-programming-budgeting-execution process, as well as the quantitative 
skills of cost estimation. 

I appreciate very much what this committee seeks do accomplish, Mr. Chairman. 

This concludes my prepared statement. 

For further information about this statement, please contact: 
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Daniel A. Nussbaum 

Graduate School of Business and Public Policy 

Department of Operations Research 

Naval Postgraduate School 

1411 Cunningham Road, GL 242 

Monterey, CA 93943 

831-656-2387 

dnussbaum@nps.edu 
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Dr. Daniel A. Nussbaum 
4160 Sunset Lane 
Pebble Beach, California 93953 
(240) 994-7715 (cell) (831) 250-6335 (home) 
dnussbaum1@gmail.co 


Synopsis: Continual involvement, over a 30-year period, supporting very senior 
management levels of government (US DoD and others) and commercial firms (for-profit 
and not-for-profit), in cost and schedule estimating and control, financial modeling, 
economics and operations research. Frequent speaker before professional groups and in 
training and educational settings. 


Education 

1964 BA, Mathematics and Economics, Columbia University. 

1971 Ph.D., Mathematics, Michigan State University. 

1973 Fellow, National Science Foundation, Econometrics and Operations Research, 
Washington State University. 

1983 National Security Management, National Defense University. 

1985 Employment of Naval Forces, US Naval War College. 

1995 Senior Officials in National Security (SONS) Fellow, Harvard University, Kennedy 
School of Government 

Employment History 

2004-present: Visiting Professor, Naval Postgraduate School, Operations Research 

Department. 

» Teach graduate level courses in cost and schedule estimating and control and other 
OR areas. 

• Lead and conduct research supporting senior government decision makers in cost 
and schedule estimating and control, cost benefit analysis, and business case 
analysis. 

• Thesis advisor and mentor for US and foreign national military officers and civilians 
attending NPS. 

• Develop strategies and plans to establish NPS presence in the National Capitol 
Region. 


1999-2004 Principal, Booz Allen Hamilton. Responsible for a broad range of cost, 
financial and economic analyses, with clients across the government and commercial 
spectrum. Developed new business opportunities for 500+ member team. 

• Major financial and management systems projects included developing financial 
and management systems within US Department of Defense and for NATO 
aspirants. 

• Led economic analyses, business case analyses, and life-cycle cost estimates to 
support cost-beneficial decisions for very large government systems within Air 
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Force, Navy, Director of Central Intelligence, and Health and Human Services 
Departments. 

1996-1999 Director, Naval Center for Cost Analysis, Office of the Assistant 
Secretary of Navy (Financial Management and Comptroller), Washington, DC. 

• Chief advisor to the Secretary of Navy on all aspects of cost and schedule 
estimating and control and single Navy point of contact for guiding, directing and 
strengthening cost estimating and analysis throughout the Department of Navy. 
Directed all Navy Independent Cost Estimates as required by Congress and senior 
Defense leadership on ships, aircraft, missiles, electronics, and automated 
information systems. Directed special, high visibility, cost and economic analyses 
for Secretary of Navy, Chief of Naval Operations and Commandant of Marine 
Corps. 

• Provided cost and schedule estimating and control policy and leadership for all 
Department of Navy commands Navy commands. Developed and implemented 
cost estimating standards, models, tools, and web and database resources for 
Navy and OSD analysts. 

• Developed first integrated cost research program to identify what needed to be 
done to improve cost estimating and analysis tools in the Departments of Navy and 
Defense. 

1987-1996 Division Head, Nava! Center for Cost Analysis; Washington, DC. 

• Headed cost engineering, aircraft and weapons divisions. Conducted Independent Cost 
Estimates and cost-benefit analyses on all naval weapons, naval aircraft and aircraft 
systems. Developed and implemented cost estimating tools. Results provided critical 
and required cost estimates to the most senior levels of decision makers in the Navy 
and Department of Defense. 

• Planned and executed the Navy's research program covering all aspects of cost and 
acquisition technologies. Provided products and results to senior leadership in Navy. 

1982-1986 Deputy Director and Acting Director, Operations Research and Cost 
Analysis Divisions, Naval Air Systems Command; Washington, DC. Planned and 
conducted cost and cost/benefit analyses on all major naval aviation systems, including 
fixed and variable wing aircraft, missiles, and automated information systems. Results 
were furnished to senior level decision makers in the Departments of Navy and Defense 
and used as the analytical basis for defensible and reasoned major capital investment 
decisions. 

1977-1981 Director, Economic Analysis and Operations Research, Headquarters, US 
Army, Europe; Heidelberg, Germany Directed headquarters’ program of mathematical 
modeling in support of resolving critical operations research and resource management 
problems, including foreign currency fluctuations, relationship of funding to military 
readiness, value engineering, capital investment decisions and total cost of operations. 
Provided high-level operations research and econometric support to senior decision 
makers across the command. Created an operations research cell for the European 
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theater to provide civilian analysts training in real world defense issues, while providing 
headquarters with greater analytic talent. 

1974-1976 Mathematician/Operations Research Analyst; U.S. Army Concepts 
Analysis Agency; Bethesda, Maryland. Designed, developed, managed and led 
operations research/systems analysis methodologies and studies. Conducted operations 
research studies in financial management, resource analysis, manpower utilization, and 
logistics. Studies were grounded in mathematical programming, econometric modeling, 
Markov processes and dynamic inventory modeling. Results were instrumental in building 
efficient and effective Army short term and long terms financial plans. 

1969-1974 Faculty member and Chairman, Department of Mathematics and 
Statistics, Saginaw Valley State University; University Center, Michigan. Developed 
an taught courses in mathematics, statistics, computer science and economics. Built 
curricula to support applied engineering program and led department initiatives in budget, 
personnel and public affairs. Recipient of University’s award for excellence. 

Professional Activities 

• Society of Cost Estimating and Analysis; Arlington, VA. 

o National President 2003-2008 
o Board of Directors 1992-present 

o Responsible for the highly regarded national training and certification 
program of an international professional organization. Provide oversight 
of content and staffing of annual training and training workshops. 
Responsible for designing and managing acquisition of a training tool 
which covers the body of knowledge in cost estimating and analysis in a 
single interactive, automated teaching tool. This syllabus and tool are 
now in the hands of over 3500 professional cost estimators around the 
world. 

• Washington Institute for Operations Research and Management Science. 

o President 
o Vice President. 

• Military Operations Research Society (MORS). 

o Board of Directors, 1988-91 
o Working Group Advisor/Chair 


Major Publications: Available upon request 
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Testimony for the House Committee on Armed Services 
- Panel on Defense Acquisition Reform - 

May 19, 2009 

Statement of John T, Dillard, Senior Lecturer, Naval Postgraduate School 

I am John T. Dillard, a member of the faculty at the Naval Postgraduate School, 
Graduate School of Business and Public Policy, Monterey, California. I’ve spent the last 
twenty-five years studying or participating in the defense acquisition management 
system. In my current capacity, I educate mid-career military officers and Department of 
Defense civilians in Defense Systems Acquisition Management, and I conduct research 
on acquisition policy and program decision-making. 

Chairman Roberts, distinguished members of the panel, I would like to thank you 
for this opportunity to discuss my thoughts on how to improve the acquisition of materiel 
in the Department of Defense. These ideas are mine and do not necessarily reflect the 
views of the Navy or the Department of Defense. 

A Complex Business Fraught With Uncertainty and Risk 

The first thing we have to acknowledge is that the acquisition of major weapon 
systems is a very complex business. Defense programs are of very large scale, with long 
product cycles, integrating advancing technologies and for many stakeholders. A plethora 
of statutes and regulations further burdens these programs with very high transaction 
costs. They are conducted as highly structured projects involving research, development, 
testing and manufacturing of materiel in the pursuit of competitive advantage in 
warfighting capability. Thus, American lives and National Security depend on these high- 
risk project outcomes. 

In projects of all types (construction, automotive, pharmaceutical, etc.), outcomes 
are measured against original estimates of cost, schedule and technical performance. 
Assessment of terminal outcomes is easy enough, but the ability to accurately forecast 
these three measures is critical, and is the greatest challenge for the DoD. Given DoD’s 
project uncertainty from technical, organizational, and procedural complexity - invention 
on a budget and schedule is the current expectation. All too often, we see DoD’s 
programs compared to those of dissimilar automotive or computer industry products, that 
are developed in annual business cycles and operate in completely different 
environments. While few would dispute the technical military superiority of fielded US 
equipment, it is easy to question whether development and procurement of our materiel 
and associated services has occurred in the most efficient and effective way. Budgets are 
huge, and the reports now frequently produced by the Government Accountability Office 
to assess major weapons programs inform us that predicted-versus-realized outcomes 
continue to diverge. There are instances of both success and failure. 

Policy “churn” With Unclear Results 

Over the last two decades, the DoD has implemented a multitude of initiatives to 
reform acquisition, focused upon both how and what it acquires. 
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- “ How " equates to “process.” Dozens of legislative and regulatory 
changes have encouraged process streamlining by standardizing processes, 
moving to electronic commerce, requiring use of commercial product 
specifications and data, extensive outsourcing of government services, credit card 
purchases and even non-traditional transaction vehicles in lieu of contracts. 

- “What equates to “requirements.” By allowing downward revisions to 
requirements as projects progress, the DoD attempts to hold unit “cost as an 
independent variable,” in order to trade-off the most significant cost-drivers; and 
more recently, to acquire systems incrementally, incorporating only the most 
mature technologies in its systems, and attaining requirements in blocks. 

There has yet to be a comprehensive assessment of these acquisition reform initiatives of 
the 1990s and early 2000s that could inform us of their individual or collective 
effectiveness. But we can still see the trends are worsening, evidenced in Nunn-McCurdy 
breaches and operational test failures cited by the GAO. 

The most recent (2008) additions to the above policy thrusts include more 
emphasis upon the front-end of projects, requiring a Department-level decision before 
materiel solutions are explored, then having at least two competitive prototypes for 
technical demonstration before advanced development begins, and using only fixed-price 
contracts for the remaining development and production efforts. 

I am in hearty agreement with the DoD’s latest reforms with regard to shortening 
product cycles by limiting project scope, and orienting decision points upon progress in 
technology maturity, design stability, and production readiness. Shorter development 
cycles should reduce some inherent complexity and make it easier to more accurately 
estimate cost and schedule. I am also a strong proponent of planning and preparation 
prior to investment, and the maintenance of project discipline via configuration control, 
staged development, and multiple sources of critical components and prototypes. These 
things are already in place in DoD’s most recent policy, along with use of contracting 
types appropriate to risk. I have seen these methods used with success firsthand to deliver 
major weapon systems on time, on budget, and to the satisfaction of technical and 
operational requirements. 1 am one who does not think that the acquisition processes are 
“broken,” though indeed our requirements generation processes may be. I feel that DoD 
seems to be largely on the right path and in synch with the GAO’s recommendations. 
However, there are reforms that could be made that would have a direct impact on costs. 

Managing Bureaucracy Instead of Programs - and Driving Up Transaction Costs 

To my continued frustration, the Department’s policies have been revised too 
frequently, and without any analysis of the results from previous revisions. Much to the 
confusion of its workforce, the Department has rewritten its series of acquisition 
instructions four times in this decade already. The constant churn in policy has left 
government and industry with an array of jargon for what should have become standard 
terminology. Worse than this, however, has been the addition of numerous and redundant 
external program reviews at various levels above the program office. Specifically, these 
are conducted at the portfolio (Program Executive Office), service or component, and 
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department levels. Oversight of the services is necessary and important, but we have over 
done it. 


My most recent collaborative research on this subject was with the National 
Research Council in 2008, commissioned to study options for reducing reviews of US Air 
Force programs to increase their cost-effectiveness and to lessen their workforce impact. 
We found that increasing numbers of external reviews, and the extensive preparation for 
them, added significantly to program transaction costs and diverted management 
resources from their core project management mission. Program Managers, who serve at 
the crucial layer of work execution, are spending less than half of their time actually 
managing their program. Much of it is instead reporting information up the chain of 
command. We counted over 88 major reviews in the typical 10-year life of a program. 

We heard of instances where portfolio level managers (Program Executive Officers), 
were assigning one Program Manager to run the program, and a twin asset dedicated to 
managing only the reviews. These costs of compliance arc not being captured, but fellow 
researchers and I believe they may be as much as 10-15% of total program costs - 
perhaps as much as would constitute a Nunn-McCurdy breach in excess program costs. 

The collective research by the GAO on defense acquisition over the period 1996 - 
2008 has concluded that the DoD usually spends more time and money than originally 
planned on weapon systems. I feel it safe to say that increasing external program reviews 
in policy changes over the same period are apparently not serving to improve these 
outcomes. The proliferation of autonomous functional area fiefdoms within the 
Department continues to increase, with each being a stovepipe of oversight expertise 
imposing unique reporting requirements, assessments and reviews. The Department 
should seek to optimize its information flows with improved synergy among its staff and 
horizontal integration of the knowledge gained from fewer reviews. Please examine the 
size of the bureaucracy we have created in that five-sided building, and how many layers 
are imposing unnecessary and costly oversight. I am a strong proponent of program 
discipline, but we have staffed the Pentagon with too many “great Americans” who do 
not add value. 

Another major challenge that has always been a Departmental constraint is the 
disconnect among the three major defense decision support systems. While the 
Requirements generation process we now call JCIDS (Joint Capabilities Integration 
System), and the project management processes we call the Defense Acquisition 
Management System are very much event driven , progressing along with combat 
contingencies, threat changes, and technological progress, etc. The funding process called 
the PPBE (Planning, Programming, Budgeting and Execution) system is very much 
calendar driven with the annual authorization and appropriation scheme that is used to 
formulate the departmental budget. And despite varying categories of funds with varying 
lifespans for obligation, this one disconnect results in unstable resource streams for 
programs, as each competes annually within the departments zero-sum pie and is subject 
to decrementing as priorities change, etc. Program managers cannot count upon a stable 
funding stream for their programs, and if they should need more time to progress than 
planned, they are incentivized to march toward failure rather than lose the funds that were 
allocated for events in the current year. The PPBE system of funds allocation prevents 
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agility in anticipating and fulfilling needs of the Combatant Commanders. In short, the 
Department can’t get out of its own way here. 

With further regard to people - the DoD’s acquisition workforce: We have seen 
in recent days legislative action that will increase badly needed government workers 
overseeing the actual acquisition work. This will help to counter almost two decades of 
reductions that have hampered the Department’s efforts to deliver materiel to the forces. 
During this same period, we have also received much less “bang for the buck” in 
payments to contractors for inherently governmental functions. The additions need to go 
beyond just the acquisition disciplines of cost estimation, systems engineering, and 
contracting. We need these people in logistics, test and evaluation, financial 
management, program management, quality assurance, etc. These new people should not 
be housed in the Pentagon, but instead where the execution of programs occurs. And with 
this ongoing institutional replenishment, we would like to see robust education and 
training as an integral piece of the DoD’s human capital development. 

With much appreciation for what this committee seeks do accomplish, Mr. Chairman, 
this concludes my prepared statement. 

For further information about this statement, please contact: 

John T. Dillard 

Senior Lecturer, Graduate School of Business and Public Policy 

Naval Postgraduate School 

555 Dyer Road, Code BPP/Dj 

Monterey, CA 93943 

831-656-2650 

idillard@nps.edu 
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RESPONSE TO QUESTION SUBMITTED BY MR. ANDREWS 

Colonel Patterson. To appreciate the context in which early C-17 cost overruns 
occurred, some background and explanation of a methodology for arriving at an esti- 
mate of how much of the cost overruns could be attributed to derived requirements 
is in order. In this case engineering changes were the predominant example of what 
I have described as derived requirements and that appear in the public record. 

The first manufactured part for the first C-17 was milled in November of 1987 
and the first squadron was declared to have Initial Operating Capability in January 
1995. The period between these events was one of significant turbulence for the C— 
17 program. The contract for the C-17 program was a fixed-price, incentive fee de- 
velopment contract, because the aircraft was to be designed using off-the-shelf tech- 
nology and was determined to be “low” risk. 

At the time of contract award, both the Government and MDC [McDonnell Doug- 
las Corporation] envisioned a program based on commercial practices, minimum 
Government involvement and concurrent development I production effort. Con- 
sequently, a fixed price incentive contract was used to match the perceived low 
risk of the program . 1 

As the program became more complex and engineering changes more frequent, 
MDC had no recourse to recoup investments in solving the engineering problems 
and meeting the cost of addressing requirements that were being added during full 
scale engineering and development than to make claims to the Government against 
the contract. 2 Though it is not feasible to determine precisely how much of the total 
cost overrun was attributable to derived requirements or engineering changes, it is 
possible to understand the relationship between the unanticipated cascade of engi- 
neering changes and increased requirements and cost overruns by virtue of the 
amount of the claim MDC believed justifiable. 

COST OVERRUNS AND ENGINEERING CHANGES 

As early as 1985 engineering changes were impacting the C-17 program costs. 
The Selected Acquisition Report for December 1985 described cost increase of 
$214.5M for engineering changes “needed for a four-pallet ramp, a combat offload 
rail system, and DoD standard avionics racks.” 3 Though these modifications were 
not in 1985 referred to as “derived requirements,” they do meet the definition used 
in the study included with the oral statement provided for the record on May 19, 
2009. 4 These were the typical derived requirements not in the original contract, but 
identified by the program management to be necessary during development and pre- 
sented as engineering changes. These changes and tbe corresponding cost increase 
occurred in the very early stages of the program, In December of 1985 the Full Scale 
Development contract had been signed. 5 The Government became alarmed following 
a 1992 Department of Defense Inspector General Report that identified a cost over- 
run of $700M on a $6.6B contract ceiling. 6 McDonnell Douglas was paying for costs 
overruns on the fixed-price contract and submitted claims for approximately $300M 
for what MDC asserted were changes in scope of its initial contract agreement 
which included “costs for engineering, development, testing and production of six C— 


1 Report of the Defense Science Board Task Force on C— 17 Review, December 1993, p 3 

2 Ibid. 

3 Selected Acquisition Report As of December 1985, p 10 

4 Derived requirements, on the other hand, are requirements that the customer has not speci- 
fied directly as a requirement but that emerge or derive from the design decisions that are 
made. Derived requirements are not capabilities that the customer specifically has identified. 
Particularly troublesome is a subset of derived requirements that fall into the category of engi- 
neering changes — those changes that improve on “good enough” and that have a combined effect 
of driving up costs and missing schedule milestones. 

5 Aeroflight, Aircraft of the World, Boeing C-17 Globemaster 111, retrieved May 27, 2009, from 
http://www.aeroflight.co.uk/types/usa/boeing/c-17/c-17.htm 

6 Battershell, A. L., The DoD C-17 versus the Boeing 777, A Comparison of Acquisition and 
Development, National Defense University, Washington, D.C. 1999, p 87—88 
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17’s and three test planes.” 7 In the end, after significant negotiations with the De- 
partment of Defense, McDonnell Douglas agreed to spend $456M in process im- 
provements and the Defense Department agreed to “provide an additional $438M 
for the program.” 

A significant portion of the cost problems centered on a cascade of engineering 
changes that typified the years and immediate several months leading up to the ne- 
gotiated settlement between MDC and the Department of Defense. The magnitude 
of the engineering changes occurring during this period was described clearly in the 
December 1993 Defense Science Board’s report as it stated, “In May 1993 alone, 
over 1,000 changes were issued directly impacting production. The backlog of engi- 
neering changes at the end of May 1993 reveals 5,800 open work authorizations.” 8 
Additionally, costly changes to the program were government-imposed. A typical ex- 
ample is the C— 17 test program which was increased from an 80 aircraft month test 
program to a “rebaselined test program of approximately 152 aircraft months” at the 
recommendation of the Defense Science Board study, with cost split evenly between 
the Government and MDC. 9 

SOME EDUCATED CONCLUSIONS 

With the data that is immediately available from the historical record it is reason- 
able to conclude that the engineering changes were significant and made an impact 
on cost, schedule and performance on the C— 17 program. Referring again to 
Battershell, of the $7.3B for the development costs and the first six aircraft in pro- 
duction Lots I and II, MDC invested approximately $1.7B of its own funds. 10 This 
amounts to about 26 percent of the original contract amount of $6.6B. MDC pro- 
posed a claim against the government of $1.2B in addition to the $438M it had re- 
ceived in the settlement 11 intended to recover what it believed to be legitimate costs 
associated with program changes a significant number of which were engineering 
changes. Though the term “significant” used above is not a precise accounting re- 
garding “how much” of the program was “attributable” to derived requirements, it 
does provide a qualitative data point that is important in evaluating opportunities 
to improve acquisition programs in the future. [See page 11.1 

o 


7 Adelson, A., Company News; McDonnell May Submit Big C-17 Bill, New York Times, Janu- 
ary 14, 1993, retrieved May 27, 2009, from http://www.nytimes.com/1993/01/14/business/com- 
pany-news-mcdonnell-may-submit-big-c-17-bill.html 

8 Report of the Defense Science Board Task Force on C-17 Review, December 1993, p 3 

9 Ibid, p 12 

10 Battershell, p 90 

11 Government Accountability Office Report, (GAO/NDIAD-94-141), Military Airlift C-17 Set- 
tlement Is Not a Good Deal, April, 1994, p 6 



